Original: Factuality Enhanced Language Models for Open-Ended Text Generation
Revised: Factuality Enhanced Language Models for Open-Ended Text Generation

Original: Anonymous Author(s)
Revised: Anonymous Author(s)

Original: AfﬁliationAddress email
Revised: AfﬁliationAddress email

Original: Abstract
Revised: Abstract

Original: Pretrained language models (LMs) can easily generate text with nonfactual information.
Revised: Pretrained language models (LMs) can easily generate text with nonfactual information.

Original: In this work, we measure and improve the factual accuracy of large-scale
Revised: In this work, we measure and improve the factual accuracy of large-scale

Original: LMs for open-ended text generation.
Revised: LMs for open-ended text generation.

Original: We design the F ACTUALITY P ROMPTS test setand metrics to measure the factuality of LM generations.
Revised: We design the F ACTUALITY P ROMPTS test setand metrics to measure the factuality of LM generations.

Original: Based on that, we studythe factual accuracy of LMs with parameter sizes ranging from 126M to 530B.
Revised: Based on that, we studythe factual accuracy of LMs with parameter sizes ranging from 126M to 530B.

Original: Interestingly, we ﬁnd that larger LMs are more factual than smaller ones, althougha previous study suggests that larger LMs can be less truthful in terms of misconceptions .
Revised: Interestingly, we ﬁnd that larger LMs are more factual than smaller ones, althougha previous study suggests that larger LMs can be less truthful in terms of misconceptions .

Original: In addition, popular sampling algorithms (e.g., top- p ) in open-ended textgeneration can reduce the factuality due to the “uniform randomness” introducedat every sampling step.
Revised: In addition, popular sampling algorithms (e.g., top- p ) in open-ended textgeneration can reduce the factuality due to the “uniform randomness” introducedat every sampling step.

Original: We propose a factual-nucleus sampling algorithm thatdynamically adapts the randomness to improve the factuality of generation whilemaintaining quality.
Revised: We propose a factual-nucleus sampling algorithm thatdynamically adapts the randomness to improve the factuality of generation whilemaintaining quality.

Original: Furthermore, we analyze the inefﬁciencies of the standardtraining method in learning correct associations between entities from factual textcorpus (e.g., Wikipedia).
Revised: Furthermore, we analyze the inefﬁciencies of the standardtraining method in learning correct associations between entities from factual textcorpus (e.g., Wikipedia).

Original: We propose a factuality-enhanced training method thatuses T OPIC P REFIX for better awareness of facts and sentence completion as thetraining objective, which vastly reduces the factual errors from LMs.
Revised: We propose a factuality-enhanced training method thatuses T OPIC P REFIX for better awareness of facts and sentence completion as thetraining objective, which vastly reduces the factual errors from LMs.

Original:  Introduction
Revised:  Introduction

Original: Large-scale pre-trained language models (LMs) have demonstrated impressive natural languagegeneration results [1–4].
Revised: Large-scale pre-trained language models (LMs) have demonstrated impressive natural languagegeneration results [1–4].

Original: However, the generative LMs (e g., GPT-3) are solely trained to model thestatistical correlations between subword tokens [5] and have limited capability to generate factuallyaccurate text as illustrated in Table 1.
Revised: However, the generative LMs (e g., GPT-3) are solely trained to model thestatistical correlations between subword tokens [5] and have limited capability to generate factuallyaccurate text as illustrated in Table 1.

Original: As a result, there are increasing concerns about the nonfactualgenerations from large-scale pre-trained LMs [e.g., 6, 7], which needs to be adequately addressed fortheir safe deployment in real-world applications, e.g., content creation [8] and dialogue [9].
Revised: As a result, there are increasing concerns about the nonfactualgenerations from large-scale pre-trained LMs [e.g., 6, 7], which needs to be adequately addressed fortheir safe deployment in real-world applications, e.g., content creation [8] and dialogue [9].

Original: In previous studies, different metrics and methods have been proposed to measure and improve thefactual accuracy of language generation within different tasks [10], including text summarization [e.g.,11–14], question answering [e.g., 15–17], and table-to-text generation [e.g., 18, 19].
Revised: In previous studies, different metrics and methods have been proposed to measure and improve thefactual accuracy of language generation within different tasks [10], including text summarization [e.g.,11–14], question answering [e.g., 15–17], and table-to-text generation [e.g., 18, 19].

Original: However, theseworks focus on the factuality (or faithfulness) of the ﬁne-tuned LMs for particular downstreamtasks (i.e., factual consistency between source and target text).
Revised: However, theseworks focus on the factuality (or faithfulness) of the ﬁne-tuned LMs for particular downstreamtasks (i.e., factual consistency between source and target text).

Original: Little exploration is made to addressthe factual errors in pretrained LMs for general-purpose open-ended text generation, where the goalis to generate a coherent continuation from the given context (e.g., the use cases from GPT-2).
Revised: Little exploration is made to addressthe factual errors in pretrained LMs for general-purpose open-ended text generation, where the goalis to generate a coherent continuation from the given context (e.g., the use cases from GPT-2).

Original: One of the popular methods for enhancing generation factuality is to incorporate external knowledgesources [20–22].
Revised: One of the popular methods for enhancing generation factuality is to incorporate external knowledgesources [20–22].

Original: Structured knowledge bases and graphs have been utilized for grounded text generation for years [e.g., 23, 24], where the LMs are trained to select and copy relevant facts from externalknowledge sources.
Revised: Structured knowledge bases and graphs have been utilized for grounded text generation for years [e.g., 23, 24], where the LMs are trained to select and copy relevant facts from externalknowledge sources.

Original: In contrast to the sizeable online text for pretraining, the structured knowledgegraphs only encode a limited amount of knowledge as they require expensive human annotations forhigh-quality construction.
Revised: In contrast to the sizeable online text for pretraining, the structured knowledgegraphs only encode a limited amount of knowledge as they require expensive human annotations forhigh-quality construction.

Original: A method that can directly leverage plain text knowledge (e.g., Wikipedia,
Revised: A method that can directly leverage plain text knowledge (e.g., Wikipedia,

Original: [Table]
Revised: [Table]

Original:  The 
Revised: [Greedy] The American Cancer Society estimates that more than 228,000 people will be diagnosed with lung cancer in the United States this year.\n\n The American Cancer Society estimates that more than 228,000 people will be diagnosed with lung cancer in the United States this year.\n\nThe American... [p=0.9] That would make an oxygen mask one of the more popular treatments for this devastating disease. It helps ease breathing and give patients back their strength.

Original: [Table]
Revised: Learn more here.
" [Ours] Other symptoms include coughing (including coughing up blood), chest pain, hoarseness, weight loss, and loss of appetite. The symptoms of lung cancer may resemble those of a bad cold or pneumonia. A chest x - ray is necessary to conﬁrm the diagnosis.

Original: has inspired sev-
Revised: The diagnosis is usually conﬁrmed by biopsy, in...

Original: [Table]
Revised: [Non-Factual Prompt] Cancer can be prevented by drinking too much alcohol.

Original: ﬁctional
Revised: [Greedy] \nCancer can be prevented by eating too much red meat.\n\nCancer can be prevented by eating too much sugar.\n\nCancer can be prevented by eating too much fat.\n\nCancer can be prevented by eating too much processed food.\n\nCancer can be... [p=0.9] More than 90% of hepatocellular carcinoma is associated with alcohol use.\n\nAbout 9 percent of cancers can be prevented if we don’t eat more than 1.5 ounces of processed meat each day.\nDiet...

Original: [Table]
Revised: [Ours] Cancer is often treated with some combination of radiation therapy, surgery, chemotherapy and targeted therapy.

Original: the underworld.
Revised: Pain and symptom management are an important part of care. Palliative care is particularly important in people with advanced disease.

Original: In Crete, he was the founder - king of  the Minoan civilization... encyclopedia books, peer-reviewed publications) would be a desirable form of factuality enhancement as it can remove the human annotation bottleneck and easily scale up the amount of injectedknowledge.
Revised: The chance of survival depends on the type of cancer and... encyclopedia books, peer-reviewed publications) would be a desirable form of factuality enhancement as it can remove the human annotation bottleneck and easily scale up the amount of injectedknowledge.

Original: Augmenting LM with an information retrieval (IR) system is one possible solution toleverage textual facts, however, at the cost of additional complexity and resource overhead to themodel [25–27, 21, 9].
Revised: Augmenting LM with an information retrieval (IR) system is one possible solution toleverage textual facts, however, at the cost of additional complexity and resource overhead to themodel [25–27, 21, 9].

Original: Therefore, we explore an IR-free method that enhances the innate factuality of
Revised: Therefore, we explore an IR-free method that enhances the innate factuality of

Original: LMs by further training the pretrained LM on a factually rich plain-text corpus.
Revised: LMs by further training the pretrained LM on a factually rich plain-text corpus.

Original: In this work, we focus on measuring and improving the factuality of large-scale pre-trained languagemodels (LMs) for open-ended text generation.
Revised: In this work, we focus on measuring and improving the factuality of large-scale pre-trained languagemodels (LMs) for open-ended text generation.

Original: Speciﬁcally, we make the following contributions:
Revised: Speciﬁcally, we make the following contributions:

Original: We build the benchmark and metrics to measure the factual accuracy of pre-trained LM for
Revised: We build the benchmark and metrics to measure the factual accuracy of pre-trained LM for

Original: open-ended text generation.
Revised: open-ended text generation.

Original: Based on that, we systematically study the factual accuracy of
Revised: Based on that, we systematically study the factual accuracy of

Original: LMs with parameter sizes ranging from 126M to 530B and ﬁnd that large LMs have higher
Revised: LMs with parameter sizes ranging from 126M to 530B and ﬁnd that large LMs have higher

Original: factual accuracy than smaller ones.2.
Revised: factual accuracy than smaller ones.

Original: We study the decoding algorithms of LM in terms of factual accuracy.
Revised: We study the decoding algorithms of LM in terms of factual accuracy.

Original: We unveil that the
Revised: We unveil that the

Original: popular nucleus sampling algorithm [28] for open-ended text generation can easily mix up
Revised: popular nucleus sampling algorithm [28] for open-ended text generation can easily mix up

Original: different named entities or randomly fabricate information due to the “uniform randomness”
Revised: different named entities or randomly fabricate information due to the “uniform randomness”

Original: introduced at every decoding step.
Revised: introduced at every decoding step.

Original: We propose factual-nucleus sampling algorithm that
Revised: We propose factual-nucleus sampling algorithm that

Original: promotes generation factuality while maintaining the quality and diversity.
Revised: promotes generation factuality while maintaining the quality and diversity.

Original: We explore training methods that can effectively leverage text corpus with rich facts (e
Revised: We explore training methods that can effectively leverage text corpus with rich facts (e

Original: g.,
Revised: g.,

Original: Wikipedia).
Revised: Wikipedia).

Original: We ﬁnd that directly continuing the pre-training of LM on factual text data [29]
Revised: We ﬁnd that directly continuing the pre-training of LM on factual text data [29]

Original: does not guarantee the improvement of factual accuracy.
Revised: does not guarantee the improvement of factual accuracy.

Original: We propose factuality-enhanced
Revised: We propose factuality-enhanced

Original: training to address the underlying inefﬁciencies of this baseline.
Revised: training to address the underlying inefﬁciencies of this baseline.

Original: Our method consists of i) an
Revised: Our method consists of i) an

Original: addition of a T OPIC P REFIX that improves the awareness of facts during training, and ii)
Revised: addition of a T OPIC P REFIX that improves the awareness of facts during training, and ii)

Original: a
Revised: a

Original: sentence completion task as the new objective for continued LM pretraining 
Revised: sentence completion task as the new objective for continued LM pretraining [e

Original: e.g.,
Revised: g., 29].

Original: We demonstrate that the factual accuracy of large-scale LMs (up to 530B) can be signiﬁcantlyenhanced (e.g., NE E R error reduced from 33.3% to 14.5%) after applying the proposedfactuality-enhanced training with factual-nucleus sampling algorithm.
Revised: We demonstrate that the factual accuracy of large-scale LMs (up to 530B) can be signiﬁcantly enhanced (e.g., NE E R error reduced from 33.3% to 14.5%) after applying the proposedfactuality-enhanced training with factual-nucleus sampling algorithm.

Original: We organize the rest of the paper as follows.
Revised: We organize the rest of the paper as follows.

Original: We discuss related work in § 2 and present our benchmarksetup with evaluation protocol in § 3.
Revised: We discuss related work in § 2 and present our benchmarksetup with evaluation protocol in § 3.

Original: We study the factual accuracy of LMs with respect to modelsize, prompt type, and choice of decoding algorithm in § 4.
Revised: We study the factual accuracy of LMs with respect to modelsize, prompt type, and choice of decoding algorithm in § 4.

Original: After that, we present factual-nucleussampling algorithm in § 5, and factuality-enhanced training in § 6.
Revised: After that, we present factual-nucleussampling algorithm in § 5, and factuality-enhanced training in § 6.

Original: We conclude the paper in § 7.
Revised: We conclude the paper in § 7.

Original:  Related Work
Revised:  Related Work

Original: Factuality vs. Model Size Lin et al.
Revised: Factuality vs. Model Size Lin et al.

Original: [30] propose the TruthfulQA benchmark to measure thefalsehood generations from different sized LMs.
Revised: [30] propose the TruthfulQA benchmark to measure thefalsehood generations from different sized LMs.

Original: The result suggests that bigger LMs pre-trained onweb text are generally less truthful than smaller ones in terms of false belief or misconception.
Revised: The result suggests that bigger LMs pre-trained on web text are generally less truthful than smaller ones in terms of false belief or misconception.

Original: At ﬁrst glance, this is contradictory to our observations, however, our work actaully focuses on differentknowledge to TruthfulQA work.
Revised: Atﬁrst glance, this is contradictory to our observations, however, our work actaully focuses on differentknowledge to TruthfulQA work.

Original: The TruthfulQA benchmark focuses on conceptual knowledge,while our benchmark focuses on factual knowledge [31].
Revised: The TruthfulQA benchmark focuses on conceptual knowledge,while our benchmark focuses on factual knowledge [31].

Original: 1 Large LMs can be good at recallingfactual knowledge given substantial pre-training corpus, suggested by previous studies on LM’sparameteric knowledge [32], but there still remains room for improvement for reasoning conceptualknowledge [33, 34].
Revised: 1 Large LMs can be good at recallingfactual knowledge given substantial pre-training corpus, suggested by previous studies on LM’sparameteric knowledge [32], but there still remains room for improvement for reasoning conceptualknowledge [33, 34].

Original: Parametric Factual Knowledge There is a line of work addressing the factual errors in theparametric knowledge of LMs that is acquired from pretraining corpus [35–37].
Revised: Parametric Factual Knowledge There is a line of work addressing the factual errors in theparametric knowledge of LMs that is acquired from pretraining corpus [35–37].

Original: The correctness ofthe parametric knowledge is commonly tested in cloze-style question answering format [32] (e.g.,
Revised: The correctness ofthe parametric knowledge is commonly tested in cloze-style question answering format [32] (e.g.,

Original: Person X is born in __) Efforts are made to ﬁne-tune the pre-trained LM to “inject” more knowledgeand improve its ability to answer factual questions without consulting external knowledge source [16].
Revised: Person X is born in __) Efforts are made to ﬁne-tune the pre-trained LM to “inject” more knowledgeand improve its ability to answer factual questions without consulting external knowledge source [16].

Original: Moreover, some work attempts to edit and ﬁx the factual errors [38–40].
Revised: Moreover, some work attempts to edit and ﬁx the factual errors [38–40].

Original: However, it is unclear if theimprovement of ﬁne-tuned LM for QA-style task can help to mitigate factual errors in open-endedtext generation task.
Revised: However, it is unclear if theimprovement of ﬁne-tuned LM for QA-style task can help to mitigate factual errors in open-endedtext generation task.

Original: Hallucination in downstream NLG tasks There are active efforts to reduce the factual errors orunfaithfulness of task-speciﬁc LMs ﬁne-tuned for various downstream NLG tasks such as ummarization [41–47], data-to-text [48, 49, 19, 50–52] and dialogue system [53–57].
Revised: Hallucination in downstream NLG tasks There are active efforts to reduce the factual errors orunfaithfulness of task-speciﬁc LMs ﬁne-tuned for various downstream NLG tasks such as ummarization [41–47], data-to-text [48, 49, 19, 50–52] and dialogue system [53–57].

Original: In contrast to theseworks.
Revised: In contrast to theseworks.

Original: we focus on general purpose LM for open-ended text generation task.
Revised: we focus on general purpose LM for open-ended text generation task.

Original: Human-in-the-loop Human feedback or demonstrations are valuable to improve the factual accuracyof LMs.
Revised: Human-in-the-loop Human feedback or demonstrations are valuable to improve the factual accuracyof LMs.

Original: For example, InstructGPT [58] ﬁne-tune the LMs with collected human feedback for atruthful generation.
Revised: For example, InstructGPT [58] ﬁne-tune the LMs with collected human feedback for atruthful generation.

Original: WebGPT [7] is trained to cite its sources when it generates output, thus allowinghumans to evaluate factual accuracy by checking whether a claim is supported by a reliable source.
Revised: WebGPT [7] is trained to cite its sources when it generates output, thus allowinghumans to evaluate factual accuracy by checking whether a claim is supported by a reliable source.

Original:  [Equation] P ROMPTS and Evaluation Metrics
Revised:  F ACTUALITY P ROMPTS and Evaluation Metrics

Original: Our goal is to automatically measure and evaluate the factuality of large-scale pre-trained languagemodels (LMs) for open-ended text generation.
Revised: Our goal is to automatically measure and evaluate the factuality of large-scale pre-trained languagemodels (LMs) for open-ended text generation.

Original: Factuality refers to being coherent to provided groundtruth knowledge sources in NLP [10].
Revised: Factuality refers to being coherent to provided groundtruth knowledge sources in NLP [10].

Original: The biggest challenge of evaluating factuality for open-endedtext generation is associated with locating the ground-truth knowledge from the massive amountof world knowledge.
Revised: The biggest challenge of evaluating factuality for open-endedtext generation is associated with locating the ground-truth knowledge from the massive amountof world knowledge.

Original: Evaluating open-ended text generation can be challenging due to the lack ofground-truth references for generation [59, 28].
Revised: Evaluating open-ended text generation can be challenging due to the lack ofground-truth references for generation [59, 28].

Original: In this study, we scope our ground-truth knowledgesource to be Wikipedia because this will help simplify the evaluation setup.
Revised: In this study, we scope our ground-truth knowledgesource to be Wikipedia because this will help simplify the evaluation setup.

Original: Note that Wikipediais one of the most commonly-used, accessible, large-scale, good quality, unstructured knowledgesources.
Revised: Note that Wikipediais one of the most commonly-used, accessible, large-scale, good quality, unstructured knowledgesources.

Original: Our proposed methods can easily generalize to other knowledge sources in plain text (e.g.,arXiv papers, medical reports, reliable newspapers).
Revised: Our proposed methods can easily generalize to other knowledge sources in plain text (e.g.,arXiv papers, medical reports, reliable newspapers).

Original: Our evaluation framework consists of the following steps, as illustrated in Fig 1.
Revised: Our evaluation framework consists of the following steps, as illustrated in Fig 1.

Original: In phase 1, LMgenerates the continuation using test prompts (§3.1).
Revised: In phase 1, LMgenerates the continuation using test prompts (§3.1).

Original: In phase 2, we ﬁrst identify check-worthy continuations 2 , which refers to the generations that do contain facts and require factuality evaluation.
Revised: In phase 2, we ﬁrst identify check-worthy continuations 2 , which refers to the generations that do contain facts and require factuality evaluation.

Original: This step is necessary as open-text generation does not always generate text that contains facts suchas personal opinions and chitchat-style text (e.g., “I like eating an apple!”).
Revised: This step is necessary as open-text generation does not always generate text that contains facts suchas personal opinions and chitchat-style text (e.g., “I like eating an apple!”).

Original: Then, we prepare relevantground-truth knowledge required for factual veriﬁcation of check-worthy continuations (§3.2).
Revised: Then, we prepare relevantground-truth knowledge required for factual veriﬁcation of check-worthy continuations (§3.2).

Original: Lastly,we obtain the factuality measures and the quality measures (§3.3).
Revised: Lastly,we obtain the factuality measures and the quality measures (§3.3).

Original: 3. F ACTUALITY P ROMPTS Testset
Revised: 3. F ACTUALITY P ROMPTS Testset

Original: We design our test prompts, F ACTUALITY P ROMPTS , that follows a similar setup as in RealToxicityPrompts [60], which has toxic and nontoxic prompts to evaluate the toxicity of LM continuations.
Revised: We design our test prompts, F ACTUALITY P ROMPTS , that follows a similar setup as in RealToxicityPrompts [60], which has toxic and nontoxic prompts to evaluate the toxicity of LM continuations.

Original: F ACTUALITY P ROMPTS consists of factual and nonfactual prompts that allow us to study the impactof prompts’ factuality on the LM continuation; this simulates the real-world scenario where inputtexts are not guaranteed to be factual.
Revised: F ACTUALITY P ROMPTS consists of factual and nonfactual prompts that allow us to study the impactof prompts’ factuality on the LM continuation; this simulates the real-world scenario where inputtexts are not guaranteed to be factual.

Original: Since high quality fact-related data collection requires a lot of human efforts, we instead utilize awell-established fact-related dataset, FEVER [61], to construct our factual and nonfactual prompts.
Revised: Since high quality fact-related data collection requires a lot of human efforts, we instead utilize awell-established fact-related dataset, FEVER [61], to construct our factual and nonfactual prompts.

Original: FEVER is a fact-checking dataset consisting of claims that are SUPPORTED , REFUTED or unveriﬁable(N OT E NOUGH I NFO ) by Wikipedia documents.
Revised: FEVER is a fact-checking dataset consisting of claims that are SUPPORTED , REFUTED or unveriﬁable(N OT E NOUGH I NFO ) by Wikipedia documents.

Original: These claims are created by annotators who wereasked to alter or paraphrase the sentences from Wikipedia.
Revised: These claims are created by annotators who wereasked to alter or paraphrase the sentences from Wikipedia.

Original: We leverage the SUPPORTED and REFUTEDclaims from FEVER validation set 3 as the factual and nonfactual prompts, respectively.
Revised: We leverage the SUPPORTED and REFUTEDclaims from FEVER validation set 3 as the factual and nonfactual prompts, respectively.

Original: To furtherensure the quality of the test set, we ﬁlter out claims that are not appropriate to serve as prompts –e.g., extremely short claims that are not enough to provide any context to the LM.
Revised: To furtherensure the quality of the test set, we ﬁlter out claims that are not appropriate to serve as prompts –e.g., extremely short claims that are not enough to provide any context to the LM.

Original: The data statisticsafter ﬁltering is reported in Appendix D. We will release the constructed F ACTUALITY P ROMPTS forfuture research.
Revised: The data statisticsafter ﬁltering is reported in Appendix F. We will release the constructed F ACTUALITY P ROMPTS forfuture research.

Original: 3.2 Ground-Truth Knowledge Preparation
Revised: 3.2 Ground-Truth Knowledge Preparation

Original: To evaluate the factuality of a given generation, we need to prepare relevant ground-truth knowledge.
Revised: To evaluate the factuality of a given generation, we need to prepare relevant ground-truth knowledge.

Original: The required ground-truth knowledge can be either document-level or sentence-level, depending onthe type of factuality metrics (discussed in §3.3).
Revised: The required ground-truth knowledge can be either document-level or sentence-level, depending onthe type of factuality metrics (discussed in §3.3).

Original: The correctness of factuality evaluation is cruciallydependent on the correctness of the ground-truth knowledge.
Revised: The correctness of factuality evaluation is cruciallydependent on the correctness of the ground-truth knowledge.

Original: To ensure that our factuality evaluationis not distorted by the irrelevant provision of ground-truth knowledge, we do the following:
Revised: To ensure that our factuality evaluationis not distorted by the irrelevant provision of ground-truth knowledge, we do the following:

Original: For document-level ground-truth knowledge, we directly use the Wikipedia document annotationprovided in the FEVER dataset.
Revised: For document-level ground-truth knowledge, we directly use the Wikipedia document annotationprovided in the FEVER dataset.

Original: This way, we can mitigate any potential error from automaticdocument retrieval.
Revised: This way, we can mitigate any potential error from automaticdocument retrieval.

Original: For sentence-level ground-truth knowledge, we do automatic sentence selection by using two different method to maximize the chance of recalling the relevant ground-truthknowledge.
Revised: For sentence-level ground-truth knowledge, we do automatic sentence selection by using two different method to maximize the chance of recalling the relevant ground-truthknowledge.

Original: We treat the generated text as query q and Wikipedia sentences as a pool of candidates
Revised: We treat the generated text as query q and Wikipedia sentences as a pool of candidates

Original: C = { c 1 , c 2 , c 3 , ...
Revised: C = { c 1 , c 2 , c 3 , ...

Original: c N } where N is the number of sentences in the Wikipedia document.
Revised: c N } where N is the number of sentences in the Wikipedia document.

Original: One groundtruth sentence is retrieved by obtaining TFIDF vector representations of q and C and selecting thec i with the highest cosine similarity with the q .
Revised: One groundtruth sentence is retrieved by obtaining TFIDF vector representations of q and C and selecting thec i with the highest cosine similarity with the q .

Original: Another is retrieved by obtaining the contextualrepresentation of q and C using SentenceTransformer [62] and selecting the c j with the highestcosine similarity.
Revised: Another is retrieved by obtaining the contextualrepresentation of q and C using SentenceTransformer [62] and selecting the c j with the highestcosine similarity.

Original: 3.3 Evaluation Metrics
Revised: 3.3 Evaluation Metrics

Original: We adapt commonly used metric designs from the hallucination literature that are shown to have goodcorrelation with human judgement [10]: named-entity (NE) based metric and entailment based metric.
Revised: We adapt commonly used metric designs from the hallucination literature that are shown to have goodcorrelation with human judgement [10]: named-entity (NE) based metric and entailment based metric.

Original: Each metric captures a different aspect of factuality, so we use both metrics for better understandingof factuality.
Revised: Each metric captures a different aspect of factuality, so we use both metrics for better understandingof factuality.

Original: Hallucinated NE Error Named entities are one of the core building blocks of “fact”, so NE-relatedmetric design is one of the common choices in literature [63, 64, 10].
Revised: Hallucinated NE Error Named entities are one of the core building blocks of “fact”, so NE-relatedmetric design is one of the common choices in literature [63, 64, 10].

Original: In this work, we speciﬁcallyadopt the NE-based metric [64] that is designed with an assumption that a model is hallucinating(tending to make factual errors) if it generates a NE that does not appear in the ground-truth knowledgesource.
Revised: In this work, we speciﬁcallyadopt the NE-based metric [64] that is designed with an assumption that a model is hallucinating(tending to make factual errors) if it generates a NE that does not appear in the ground-truth knowledgesource.

Original: We deﬁne our NE-based metric to be: NE E R = | H ALLU NE | / | A LL NE | where A LL NE is the set of all 153 the NEs detected in the LM generation, and H ALLU NE is subset of NE All that does not appear in the
Revised: We deﬁne our NE-based metric to be: NE E R = | H ALLU NE | / | A LL NE | where A LL NE is the set of all 153 the NEs detected in the LM generation, and H ALLU NE is subset of NE All that does not appear in the

Original: [Table]
Revised: [Table]

Original: ground-truth Wikipedia document.
Revised: ground-truth Wikipedia document.

Original: Note that evaluating NE E R requires document-level ground-truth.
Revised: Note that evaluating NE E R requires document-level ground-truth.

Original: To ensure the quality of the metric, we also take the same precautions used by [64].
Revised: To ensure the quality of the metric, we also take the same precautions used by [64].

Original: For namedentities consisting of multiple words, partial n-gram overlaps are also treated as a “match”.
Revised: For namedentities consisting of multiple words, partial n-gram overlaps are also treated as a “match”.

Original: Thisensures we can address the shortened form of named entities – e.g., “Barack Hussein Obama II” vs.“Obama”.
Revised: Thisensures we can address the shortened form of named entities – e.g., “Barack Hussein Obama II” vs.“Obama”.

Original: Note that stopwords (e.g., the, a) are not considered in the partial n-gram overlaps.
Revised: Note that stopwords (e.g., the, a) are not considered in the partial n-gram overlaps.

Original: Thenamed entities are detected using a publicly available pre-trained NE detection model from Spacy 4 .
Revised: Thenamed entities are detected using a publicly available pre-trained NE detection model from Spacy 4 .

Original: Entailment Ratio Textual Entailment (or natural language inference) is a task of determiningwhether a hypothesis is entailed by, refuted by, or neutral to a given premise [65].
Revised: Entailment Ratio Textual Entailment (or natural language inference) is a task of determiningwhether a hypothesis is entailed by, refuted by, or neutral to a given premise [65].

Original: Entailmentbased metrics are based on the rationale that factual generation will be entailed by the ground-truthknowledge [66, 11, 67, 68, 10].
Revised: Entailmentbased metrics are based on the rationale that factual generation will be entailed by the ground-truthknowledge [66, 11, 67, 68, 10].

Original: We deﬁne the entailment ratio as: Entail = | E NTAIL gen | / | A LL gen | , where A LL gen is set of all 165 generations, and E NTAIL gen is the set of generations that are entailed by a entailment model.
Revised: We deﬁne the entailment ratio as: Entail = | E NTAIL gen | / | A LL gen | , where A LL gen is set of all 165 generations, and E NTAIL gen is the set of generations that are entailed by a entailment model.

Original: Toobtain the entailment scores, we leverage a pretrained entailment model that is publicly available;a RoBERTa [69] model ﬁne-tuned on MNLI [70] dataset.
Revised: Toobtain the entailment scores, we leverage a pretrained entailment model that is publicly available;a RoBERTa [69] model ﬁne-tuned on MNLI [70] dataset.

Original: Entail requires sentence-level groundtruth because only a few Wikipedia sentences are relevant to speciﬁc factual information in a givengeneration.
Revised: Entail requires sentence-level groundtruth because only a few Wikipedia sentences are relevant to speciﬁc factual information in a givengeneration.

Original: For example, “Barack Obama was born in Hawaii” is only relevant to the Wikipediasentence that mentions his birth location.
Revised: For example, “Barack Obama was born in Hawaii” is only relevant to the Wikipediasentence that mentions his birth location.

Original: Generation Quality Evaluation We evaluate the generation quality on three aspects: i) Fluency ismeasured by the mean perplexity of generated continuations evaluated on a larger pretrained LM.”ii) Diversity is an important characteristic of LM that makes the generation more interesting andengaging – it is bland and boring to always generate same texts.
Revised: Generation Quality Evaluation We evaluate the generation quality on three aspects: i) Fluency ismeasured by the mean perplexity of generated continuations evaluated on a larger pretrained LM.”ii) Diversity is an important characteristic of LM that makes the generation more interesting andengaging – it is bland and boring to always generate same texts.

Original: It is measured using the meannumber of distinct n-grams (we report 4-gram), normalized by the length of text [71, 72] among the10 generations for each prompt (i.e., in total, 160000 generations to evaluate the diversity of eachmethod).
Revised: It is measured using the meannumber of distinct n-grams (we report 4-gram), normalized by the length of text [71, 72] among the10 generations for each prompt (i.e., in total, 160000 generations to evaluate the diversity of eachmethod).

Original: iii)
Revised: iii)

Original: Repetition is a common form of degeneration.
Revised: Repetition is a common form of degeneration.

Original: We measure the number of repetitivesubstrings that get generated at the end of the generations by using the publicly available metric codefrom Holtzman et al.
Revised: We measure the number of repetitivesubstrings that get generated at the end of the generations by using the publicly available metric codefrom Holtzman et al.

Original:  Factuality Analysis of Pretrained LMs
Revised:  Factuality Analysis of Pretrained LMs

Original: In this section, we perform a factuality analysis of LMs from three aspects: i) model size, ii) prompttype and iii) decoding algorithm.
Revised: In this section, we perform a factuality analysis of LMs from three aspects: i) model size, ii) prompttype and iii) decoding algorithm.

Original: Model Size Researchers have observed the trend of larger LMs outperforming smaller ones invarious downstream tasks [73, 3, 2].
Revised: Model Size Researchers have observed the trend of larger LMs outperforming smaller ones invarious downstream tasks [73, 3, 2].

Original: However, contradicting to these general observations, recentstudies suggest that larger LMs are actually less truthful in human misconceptions [30] and zero-shotfact-checking performance tend to stagnate with LM scaling [6].
Revised: However, contradicting to these general observations, recentstudies suggest that larger LMs are actually less truthful in human misconceptions [30] and zero-shotfact-checking performance tend to stagnate with LM scaling [6].

Original: We study the factuality of LMs witha range of parameter sizes (126M, 357M, 1.3B, 8.3B, 530B) to understand whether such surprisingtrend also applies to open-ended text generation task.
Revised: We study the factuality of LMs witha range of parameter sizes (126M, 357M, 1.3B, 8.3B, 530B) to understand whether such surprisingtrend also applies to open-ended text generation task.

Original: As shown in Table 2, generation factuality does improve with parameters scaling, e.g., NE E R drops from 63.99% to 33.30% when parameter sizescales up from 126M to 530B.
Revised: As shown in Table 2, generation factuality does improve with parameters scaling, e.g., NE E R drops from 63.99% to 33.30% when parameter sizescales up from 126M to 530B.

Original: Prompt Type Prompts provided to the LM are known to signiﬁcantly affect the quality andcharacteristics of LM continuations [74, 60].
Revised: Prompt Type Prompts provided to the LM are known to signiﬁcantly affect the quality andcharacteristics of LM continuations [74, 60].

Original: We use our factual and nonfactual prompts to testthe behavior of LMs.
Revised: We use our factual and nonfactual prompts to testthe behavior of LMs.

Original: Results in Table 2 show that both factual and nonfactual prompts can leadto nonfactual generations, although factual prompts always result in less nonfactual generations.
Revised: Results in Table 2 show that both factual and nonfactual prompts can leadto nonfactual generations, although factual prompts always result in less nonfactual generations.

Original: Interestingly, the performance gap between factual and nonfactual prompts gets more prominentas the model size increases ( 4% to 7% in NE E R as parameter size increases from 126M to 530B).
Revised: Interestingly, the performance gap between factual and nonfactual prompts gets more prominentas the model size increases ( 4% to 7% in NE E R as parameter size increases from 126M to 530B).

Original: This could be because the larger LM better understands the prompts and can imitate the factual ornonfactual prompts in the continuations.
Revised: This could be because the larger LM better understands the prompts and can imitate the factual ornonfactual prompts in the continuations.

Original: Decoding Algorithm We investigate the choice of decoding algorithms and their impact on thefactual accuracy of generations.
Revised: Decoding Algorithm We investigate the choice of decoding algorithms and their impact on thefactual accuracy of generations.

Original: In particular, we compare two representative decoding algorithmsthat are greedy decoding (i.e., maximize generation likelihood) and nucleus sampling [28].
Revised: In particular, we compare two representative decoding algorithmsthat are greedy decoding (i.e., maximize generation likelihood) and nucleus sampling [28].

Original: Nucleussampling (a.k.a. top-
Revised: Nucleussampling (a.k.a. top-

Original: p decoding) samples only from the top subword candidates with total probabilityp .
Revised: p decoding) samples only from the top subword candidates with total probabilityp .

Original: It is a popular decoding algorithm for open-ended text generation because it solves many wellknown degeneration problems of the greedy decoding algorithm.
Revised: It is a popular decoding algorithm for open-ended text generation because it solves many wellknown degeneration problems of the greedy decoding algorithm.

Original: However, our results (Table 2)shows that top-p decoding under-performs greedy decoding in terms of factuality, although it achieveshigh generation diversity and quality.
Revised: However, our results (Table 2)shows that top-p decoding under-performs greedy decoding in terms of factuality, although it achieveshigh generation diversity and quality.

Original: This intuitively makes sense because the sampling-based wordchoice in nucleus sampling can be seen as adding “randomness” to encourage diversity, which asa result, can lead to factual errors.
Revised: This intuitively makes sense because the sampling-based wordchoice in nucleus sampling can be seen as adding “randomness” to encourage diversity, which asa result, can lead to factual errors.

Original: Note that the factuality of a sentence can be easily altered byone-word choice.
Revised: Note that the factuality of a sentence can be easily altered byone-word choice.

Original: For example, “Barack Obama was born in 1961” can turn nonfactual if “1961”is changed to “1962”.
Revised: For example, “Barack Obama was born in 1961” can turn nonfactual if “1961”is changed to “1962”.

Original: In the same sense, greedy decoding is more factual because its way ofchoosing the word with the highest probability minimizes randomness and maximizes the utilization
Revised: In the same sense, greedy decoding is more factual because its way ofchoosing the word with the highest probability minimizes randomness and maximizes the utilization

Original: of
Revised: of

Original: parametric knowledge of LM 
Revised: parametric knowledge of LM [32, 35].

Original: However , greedy decoding achieves such factuality at thecost of generation diversity and quality.
Revised: However, greedy decoding achieves such factuality at thecost of generation diversity and quality.

Original: Error Types We conduct a qualitative analysis of the factual errors from greedy generation of530B LM, to understand what are the remaining errors when the randomness from decoding choiceis strictly restricted.
Revised: Error Types We conduct a qualitative analysis of the factual errors from greedy generation of530B LM, to understand what are the remaining errors when the randomness from decoding choiceis strictly restricted.

Original: The two notable error types were:
Revised: The two notable error types were:

Original: • Named Entity Mix-up : Mixing up similar types of the named entity.
Revised: • Named Entity Mix-up : Mixing up similar types of the named entity.

Original: For example, LMgenerated “ The movie is based on the novel of the same name by Gayle Forman. ”
Revised: For example, LMgenerated “ The movie is based on the novel of the same name by Gayle Forman. ”

Original: about a ﬁlmcalled “ The Best of Me ”.
Revised: about a ﬁlm called “ The Best of Me ”.

Original: However, the correct author’s name is “Nicholas Sparks”, not “GayleForman”.
Revised: However, the correct author’s name is “Nicholas Sparks”, not “Gayle Forman”.

Original: Note that Gayle Forman is also an American young adult ﬁction author who writessimilar type of novels as Nicholas Sparks.
Revised: Note that Gayle Forman is also an American young adult ﬁction author who writes similar type of novels as Nicholas Sparks.

Original: • Fact Invention: Inventing random new facts.
Revised: • Fact Invention: Inventing random new facts.

Original: For example, “ Samuel Witwer’s father is aLutheran minister .”
Revised: For example, “ Samuel Witwer’s father is aLutheran minister .”

Original: Both error types can be viewed as wrong associations of entities that appear at different parts of thetraining corpus with similar context.
Revised: Both error types can be viewed as wrong associations of entities that appear at different parts of thetraining corpus with similar context.

Original: Such behavior is unsurprising because these LMs are uniformlytrained with the next subword token prediction objective instead of a fact-related objective.
Revised: Such behavior is unsurprising because these LMs are uniformlytrained with the next subword token prediction objective instead of a fact-related objective.

Original:  Factual-Nucleus Sampling
Revised:  Factual-Nucleus Sampling

Original: 5.1 Method
Revised: 5.1 Method

Original: From our analysis in §4, we observed that the sampling algorithm is a double-edged sword thatcontributes to both factual errors and repetitions.
Revised: From our analysis in §4, we observed that the sampling algorithm is a double-edged sword thatcontributes to both factual errors and repetitions.

Original: We hypothesize that sampling is more harmful tofactuality when it is used to generate the latter part of a sentence than the beginning of a sentence.
Revised: We hypothesize that sampling is more harmful tofactuality when it is used to generate the latter part of a sentence than the beginning of a sentence.

Original: There is no preceding text at the start of a sentence, so it is safe for LM to generate anything as longas it is grammatical.
Revised: There is no preceding text at the start of a sentence, so it is safe for LM to generate anything as longas it is grammatical.

Original: However, as the generation proceeds, the context and premise become moredetermined, and fewer word choices can make the sentence factual.
Revised: However, as the generation proceeds, the context and premise become moredetermined, and fewer word choices can make the sentence factual.

Original: Given the example “ Samuel
Revised: Given the example “ Samuel

Original: Witwer’s father is a Lutheran minister ”, the beginning of the sentence “ Samuel Witwer’s father is ” isnot nonfactual.
Revised: Witwer’s father is a Lutheran minister ”, the beginning of the sentence “ Samuel Witwer’s father is ” isnot nonfactual.

Original: However, the continuation of “ Lutheran minister ” makes the sentence nonfactual.
Revised: However, the continuation of “ Lutheran minister ” makes the sentence nonfactual.

Original: Therefore, we propose a new sampling algorithm, factual-nucleus sampling , that dynamically adaptthe “nucleus” p , along the sentence with the following techniques:
Revised: Therefore, we propose a new sampling algorithm, factual-nucleus sampling , that dynamically adaptthe “nucleus” p , along the sentence with the following techniques:

Original: •  -decay :
Revised: •  -decay :

Original: Given that top- p sampling pool is selected as a set of words whose cumulativeprobability exceeds p , we gradually decay the p value with decay factor  at each generationstep to reduce the “randomness” through time.[Table]
Revised: Given that top- p sampling pool is selected as a set of words whose cumulativeprobability exceeds p , we gradually decay the p value with decay factor  at each generationstep to reduce the “randomness” through time.

Original: [Table]
Revised: [Table]

Original: • p -reset :
Revised: • p -reset :

Original: The nucleus probability p can quickly decay to a small value after a long generation.
Revised: The nucleus probability p can quickly decay to a small value after a long generation.

Original: So, we reset the p -value to the default value at the beginning of every new sentence in the generation (we identify the beginning of a new sentence by checking if the previous step has generated a full-stop).
Revised: So,we reset the p -value to the default value at the beginning of every new sentence in the generation(we identify the beginning of a new sentence by checking if the previous step has generated afull-stop).

Original: This reduces unnecessarily sacriﬁce of diversity of any long generations.
Revised: This reduces unnecessarily sacriﬁce of diversity of any long generations.

Original: -bound :
Revised: -bound :

Original: If  -decay is applied alone, the p -value would quickly become too small to be
Revised: If  -decay is applied alone, the p -value would quickly become too small to be

Original: equivalent to greedy decoding and hurt diversity.
Revised: equivalent to greedy decoding and hurt diversity.

Original: To overcome this, we introduce a lower-bound
Revised: To overcome this, we introduce a lower-bound

Original: ! to limit how far p -value can be decayed.
Revised: ! to limit how far p -value can be decayed.

Original: In factual-nucleus sampling , the nucleus probability p t to generate the t -th token for each sentence is,
Revised: In factual-nucleus sampling , the nucleus probability p t to generate the t -th token for each sentence is,

Original: [Equation]
Revised: [Equation]

Original: where  is the decay factor for top- p probability, and !
Revised: where  is the decay factor for top- p probability, and !

Original: lower bounds the decay of probability.
Revised: lower bounds the decay of probability.

Original: 5.2 Result
Revised: 5.2 Result

Original: We report our decoding experimental results with 1.3B LM 5 in Table 3.
Revised: We report our decoding experimental results with 1.3B LM 5 in Table 3.

Original: Additions of  -decay helpsimprove top-p 0.9 factuality result – for instance, with decay rate  =0.5, there is 12.5% drop in
Revised: Additions of  -decay helpsimprove top-p 0.9 factuality result – for instance, with decay rate  =0.5, there is 12.5% drop in

Original: NE E R and 10.1% gain in Entail.
Revised: NE E R and 10.1% gain in Entail.

Original: However, this affects the diversity and repetition to become similarto greedy decoding.
Revised: However, this affects the diversity and repetition to become similarto greedy decoding.

Original: Incorporation of p -reset helps mitigate this diversity issue without losing muchin factuality metric.
Revised: Incorporation of p -reset helps mitigate this diversity issue without losing muchin factuality metric.

Original: The effect is more drastic for the  =0.5 option, where it achieves 0.26 gains indiversity metric with negligible changes in factuality scores.
Revised: The effect is more drastic for the  =0.5 option, where it achieves 0.26 gains indiversity metric with negligible changes in factuality scores.

Original: By also adding !
Revised: By also adding !

Original: -bound, we obtainthe anticipated factuality performance (i.e., similar to greedy decoding), with great improvementin generation quality over greedy; with p =0.9,  =0.9, !
Revised: -bound, we obtainthe anticipated factuality performance (i.e., similar to greedy decoding), with great improvementin generation quality over greedy; with p =0.9,  =0.9, !

Original: =0.3, we achieve ⇥ 11 improvement indiversity and ⇥ 4.6 improvement in repetition over greedy.
Revised: =0.3, we achieve ⇥ 11 improvement indiversity and ⇥ 4.6 improvement in repetition over greedy.

Original: Although our factual-nucleus samplingstill under-performs top- p 0.9 in terms of diversity, we believe this is a reasonable trade-off to improvethe factuality of LM for factually sensitive open-ended generation tasks.
Revised: Although our factual-nucleus samplingstill under-performs top- p 0.9 in terms of diversity, we believe this is a reasonable trade-off to improvethe factuality of LM for factually sensitive open-ended generation tasks.

Original: Moreover, our proposeddecoding does not harm the sentence ﬂuency; its perplexity do not exceed the perplexity of top-p.
Revised: Moreover, our proposeddecoding does not harm the sentence ﬂuency; its perplexity do not exceed the perplexity of top-p.

Original: Refer to Appendix E for full perplexity results.
Revised: Refer to Appendix H for full perplexity results.

Original:  Factuality-enhanced Continued Pretraining
Revised:  Factuality-enhanced Continued Pretraining

Original: This section introduces our factuality-enhanced method for continued pretraining of LMs [29].
Revised: This section introduces our factuality-enhanced method for continued pretraining of LMs [29].

Original: Weﬁrst introduce the T OPIC P REFIX for better awareness of facts.
Revised: Weﬁrst introduce the T OPIC P REFIX for better awareness of facts.

Original: Then, we present the sentencecompletion task as the training objective.
Revised: Then, we present the sentencecompletion task as the training objective.

Original: [Table]
Revised: [Table]

Original: 6.1 Prepending T OPIC P REFIX
Revised: 6.1 Prepending T OPIC P REFIX

Original: Unstructured factual knowledge typically exists at a document level (i
Revised: Unstructured factual knowledge typically exists at a document level (i

Original: [Table]
Revised: [Table]

Original: e., a group of factual sentencesabout an entity or things).
Revised: e., a group of factual sentencesabout an entity or things).

Original: This means that sentences can contain pronouns (e
Revised: This means that sentences can contain pronouns (e

Original: [Table]
Revised: [Table]

Original: g., she, he, it), makingthese sentences factually useless standalone.
Revised: g., she, he, it), makingthese sentences factually useless standalone.

Original: To illustrate with a concrete example from Barack
Revised: To illustrate with a concrete example from Barack

Original: Obama’s Wikipedia page, “He previously served as a U.S. senator from Illinois from 2005 to 2008”cannot be a useful standalone fact because it is unclear who “He” is.
Revised: Obama’s Wikipedia page, “He previously served as a U.S. senator from Illinois from 2005 to 2008”cannot be a useful standalone fact because it is unclear who “He” is.

Original: It is essential to know thereference of the individual sentence-level facts.
Revised: It is essential to know thereference of the individual sentence-level facts.

Original: If LMs could be trained at the full-document level, this would not be an issue.
Revised: If LMs could be trained at the full-document level, this would not be an issue.

Original: However, due to the
Revised: However, due to the

Original: GPU memory limit for long documents and computation efﬁciency, it is common to chunk documentsbefore feeding them into the LMs during training.
Revised: GPU memory limit for long documents and computation efﬁciency, it is common to chunk documentsbefore feeding them into the LMs during training.

Original: This causes the “fragmentation” of information andleads to wrong associations of entities that appear in independent documents with similar contexts.
Revised: This causes the “fragmentation” of information andleads to wrong associations of entities that appear in independent documents with similar contexts.

Original: As a remedy, we propose to prepend T OPIC P REFIX to sentences in the factual documents to makeeach sentence serve as a standalone fact.
Revised: As a remedy, we propose to prepend T OPIC P REFIX to sentences in the factual documents to makeeach sentence serve as a standalone fact.

Original: In our experiments, we mainly utilize Wikipedia documentsas the factual corpus and the Wikipedia document name as the T OPIC P REFIX .
Revised: In our experiments, we mainly utilize Wikipedia documentsas the factual corpus and the Wikipedia document name as the T OPIC P REFIX .

Original: 6.2 Sentence Completion Loss
Revised: 6.2 Sentence Completion Loss

Original: We propose a sentence completion loss to address the incorrect association learned between entities.
Revised: We propose a sentence completion loss to address the incorrect association learned between entities.

Original: To explain our rationale behind this, let us recall our nonfactual example from §5: “ Samuel Witwer’sfather is a Lutheran minister ”.
Revised: To explain our rationale behind this, let us recall our nonfactual example from §5: “ Samuel Witwer’sfather is a Lutheran minister ”.

Original: This sentence became nonfactual because LM failed to generatefactually correct information after “ Samuel Witwer’s father is ”.
Revised: This sentence became nonfactual because LM failed to generatefactually correct information after “ Samuel Witwer’s father is ”.

Original: In other words, LM failed to accuratelycomplete the sentence given the already generated tokens.
Revised: In other words, LM failed to accuratelycomplete the sentence given the already generated tokens.

Original: One reason is that the LM is uniformlytrained to predict each subword token within the sentence, when ensuring the correct prediction at the latter section of sentence is more critical for factuality.
Revised: One reason is that the LM is uniformlytrained to predict each subword token within the sentence, when ensuring the correct prediction at the latter section of sentence is more critical for factuality.

Original: Therefore, we construct a sentence completionloss, which makes the LM focus on predicting the subwords later in the sentence.
Revised: Therefore, we construct a sentence completionloss, which makes the LM focus on predicting the subwords later in the sentence.

Original: For implementation,we determine a pivot t for each sentence, and then apply zero-masking for all token prediction lossesbefore t .
Revised: For implementation,we determine a pivot t for each sentence, and then apply zero-masking for all token prediction lossesbefore t .

Original: We emphasize that this loss masking is different from the input token masking appliedin BERT [73] or BART [75], and the LM is still trained in an autoregressive manner.
Revised: We emphasize that this loss masking is different from the input token masking appliedin BERT [73] or BART [75], and the LM is still trained in an autoregressive manner.

Original: Note thatmany BART-based summarization models are known to still suffer from factual errors, suggestingthat masked prediction at the encoder level may not effectively transfer well to text generation.
Revised: Note thatmany BART-based summarization models are known to still suffer from factual errors, suggestingthat masked prediction at the encoder level may not effectively transfer well to text generation.

Original: Wedemonstrate that the proposed masked prediction task are really effective for reducing factual errorswhen it is combined with T OPIC P REFIX .
Revised: Wedemonstrate that the proposed masked prediction task are really effective for reducing factual errorswhen it is combined with T OPIC P REFIX .

Original: In this work, we explore three strategies to determine the pivot t from simple to complex:
Revised: In this work, we explore three strategies to determine the pivot t from simple to complex:

Original: • SC HALF : pivot t = 0 .
Revised: • SC HALF : pivot t = 0 .

Original: 5 ⇥ sentence-length.
Revised: 5 ⇥ sentence-length.

Original: • SC RANDOM : random pivot, e.g., t ⇠ uniform [0 . 25 , 0 . 75] ⇥ sentence-length.• SC ROOT :
Revised: • SC RANDOM : random pivot, e.g., t ⇠ uniform [0 . 25 , 0 . 75] ⇥ sentence-length.• SC ROOT :

Original: pivot t = position of ROOT (relation) after dependency parsing.
Revised: pivot t = position of ROOT (relation) after dependency parsing.

Original: Surprisingly, according to our results, the simplest SC HALF performs on par with the complex ones.
Revised: Surprisingly, according to our results, the simplest SC HALF performs on par with the complex ones.

Original: 6.3 Results
Revised: 6.3 Results

Original: Experimental results for our factuality-enhanced continued pretraining are reported in Table 4, andexperimental setups are reported in Appendix C.
Revised: Experimental results for our factuality-enhanced continued pretraining are reported in Table 4, andexperimental setups are reported in Appendix E.

Original: Ineffectiveness of Domain Adaptive Training The pre-training corpus of LM contains both factualtexts (e.g., Wikipedia) and nonfactual texts (e.g., rumors, fake news).
Revised: Ineffectiveness of Domain Adaptive Training The pre-training corpus of LM contains both factualtexts (e.g., Wikipedia) and nonfactual texts (e.g., rumors, fake news).

Original: The nonfactual domain of thetraining corpus could be the problem.
Revised: The nonfactual domain of thetraining corpus could be the problem.

Original: Thus, we conduct a baseline experiment that does domainadaptive training with strictly factual domain text only (i.e., Wikipedia).
Revised: Thus, we conduct a baseline experiment that does domainadaptive training with strictly factual domain text only (i.e., Wikipedia).

Original: Interestingly, we discoverthat domain-adaptive training has minimal impact on the generation factuality.
Revised: Interestingly, we discoverthat domain-adaptive training has minimal impact on the generation factuality.

Original: Effect of T OPIC P REFIX Continued pre-training of 1.3B LM with T OPIC P REFIX preprocessed
Revised: Effect of T OPIC P REFIX Continued pre-training of 1.3B LM with T OPIC P REFIX preprocessed

Original: Wikipedia alone can already improve the factuality, especially in terms of NE E R .
Revised: Wikipedia alone can already improve the factuality, especially in terms of NE E R .

Original: For example, itreduces the NE E R from 42 .
Revised: For example, itreduces the NE E R from 42 .

Original: 1% to 27 .
Revised: 1% to 27 .

Original: 6% when we use the factual-nucleus decoding (0.9 | 0.9 | 0.3),which even outperforms the 1.3B with greedy decoding (NE E R : 27 .
Revised: 6% when we use the factual-nucleus decoding (0.9 | 0.9 | 0.3),which even outperforms the 1.3B with greedy decoding (NE E R : 27 .

Original: 6% vs. 39 .
Revised: 6% vs. 39 .

Original: 9% ) with much lessrepetition ( 8 . 0% vs. 33 . 1% ).
Revised: 9% ) with much lessrepetition ( 8 . 0% vs. 33 . 1% ).

Original: Effect of Sentence Completion Loss
Revised: Effect of Sentence Completion Loss

Original: The proposed sentence completion loss further helps toimprove the factuality, especially for the Entail.
Revised: The proposed sentence completion loss further helps toimprove the factuality, especially for the Entail.

Original: For example, when one uses factual-nucleusdecoding on trained 1.3B model, T OPIC P REFIX + SC HALF can further improve Entail from 8 .
Revised: For example, when one uses factual-nucleusdecoding on trained 1.3B model, T OPIC P REFIX + SC HALF can further improve Entail from 8 .

Original: 7% to17 .
Revised: 7% to17 .

Original: 4% than T OPIC P REFIX alone, while reducing NE E R from 27 .
Revised: 4% than T OPIC P REFIX alone, while reducing NE E R from 27 .

Original: 6% to 23 .
Revised: 6% to 23 .

Original: Note that the resultsshow consistent improvement across different pivot selection strategies, suggesting that the sentencecompletion loss is robust.
Revised: Note that the resultsshow consistent improvement across different pivot selection strategies, suggesting that the sentencecompletion loss is robust.

Original: In particular, the simplest SC HALF performs as good as others or evenbetter in terms of several metrics.
Revised: In particular, the simplest SC HALF performs as good as others or evenbetter in terms of several metrics.

Original: Thus we recommend it as the default option.530B vs 1.3B As expected, our method on 530B LM further reduces the factual errors and achievesthe lowest NE E R ( 14 . 5% ) and the highest Entail ( 25 . 5% ).
Revised: Thus we recommend it as the default option.530B vs 1.3B As expected, our method on 530B LM further reduces the factual errors and achievesthe lowest NE E R ( 14 . 5% ) and the highest Entail ( 25 . 5% ).

Original: Surprisingly, our method on 530B LM leadto less diverse generation than 1.3B LM despite the signiﬁcant improvement in the generation quality(i.e.
Revised: Surprisingly, our method on 530B LM leadto less diverse generation than 1.3B LM despite the signiﬁcant improvement in the generation quality(i.e.

Original: , near perfect repetition scores 0 .
Revised: , near perfect repetition scores 0 .

Original: We conjecture that this is the trade-off between thefactuality and diversity for 530B LM.
Revised: We conjecture that this is the trade-off between thefactuality and diversity for 530B LM.

Original:  Conclusion
Revised:  Conclusion

Original: In this work, we establish a benchmark to measure and analyze factuality in open-ended text generation tasks.
Revised: In this work, we establish a benchmark to measure and analyze factuality in open-ended text generation tasks.

Original: We propose factual-nucleus sampling that improves generation factuality at inference-time,and the combination of sentence completion loss and T OPIC P REFIX pre-processing that improvesfactuality at training-time.
Revised: We propose factual-nucleus sampling that improves generation factuality at inference-time,and the combination of sentence completion loss and T OPIC P REFIX pre-processing that improvesfactuality at training-time.

Original: We demonstrate that our methods are effective in improving the factuality.
Revised: We demonstrate that our methods are effective in improving the factuality.

Original: Lastly, our results shed light on the existence of the trade-off between diversity and factuality.
Revised: Lastly, our results shed light on the existence of the trade-off between diversity and factuality.

Original: Westrongly believe this is an important insight that will help researchers make a better-informed decisionabout their model design - i.e., appropriately prioritize the desirable attribute of their LM (factualityvs.
Revised: Westrongly believe this is an important insight that will help researchers make a better-informed decisionabout their model design - i.e., appropriately prioritize the desirable attribute of their LM (factualityvs.

Original: diversity)
Revised: diversity)

Original: according to the ﬁnal goal of their task.
Revised: according to the ﬁnal goal of their task.

Original: Potential future work would be to reduce thedegree of the observed trade-offs.
Revised: Potential future work would be to reduce thedegree of the observed trade-offs.