Original: T HE C URIOUS C ASE OF N EURAL T EXT De GENERATION
Revised: T HE C URIOUS C ASE OF N EURAL T EXT De GENERATION

Original: Anonymous authors

Original: Paper under double-blind review

Original: A BSTRACT
Revised: A BSTRACT

Original: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story).
Revised: Despite considerable advances in neural language modeling, it remains an open question what the best decoding strategy is for text generation from a language model (e.g. to generate a story).

Original: The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops.
Revised: The counter-intuitive empirical observation is that even though the use of likelihood as training objective leads to high quality models for a broad range of language understanding tasks, maximization-based decoding methods such as beam search lead to degeneration — output text that is bland, incoherent, or gets stuck in repetitive loops.

Original: To address this we propose Nucleus Sampling , a simple but effective method to draw considerably higher quality text out of neural language models than previous decoding strategies.
Revised: To address this we propose Nucleus Sampling , a simple but effective method to draw considerably higher quality text out of neural language models than previous decoding strategies.

Original: Our approach avoids text de generation by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass.
Revised: Our approach avoids text de generation by truncating the unreliable tail of the probability distribution, sampling from the dynamic nucleus of tokens containing the vast majority of the probability mass.

Original: To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.
Revised: To properly examine current maximization-based and stochastic decoding methods, we compare generations from each of these methods to the distribution of human text along several axes such as likelihood, diversity, and repetition.

Original: Our results show that (1) maximization is an inappropriate decoding objective for openended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is  the best available decoding strategy for generating longform text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.
Revised: Our results show that (1) maximization is an inappropriate decoding objective for openended text generation, (2) the probability distributions of the best current language models have an unreliable tail which needs to be truncated during generation and (3) Nucleus Sampling is currently the best available decoding strategy for generating long-form text that is both high-quality — as measured by human evaluation — and as diverse as human-written text.

Original: Context :
Revised: Context :

Original: In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains.
Revised: In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains.

Original: Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
Revised: Even more surprising to the researchers was the fact that the unicorns spoke perfect English.

Original: Beam Search, b =100 : "The unicorns were very intelligent, and they were very intelligent," said Dr. David S. Siegel, a professor ofanthropology at the University of 
Revised: Beam Search, b =32 : "The study, published in the Proceedings of the National Academy of Sciences of the United States of America (PNAS), was conducted by researchers from the

Original: California, Berkeley.
Revised: Universidad Nacional Autónoma de México (UNAM) and the Universidad Nacional Autónoma de México (UNAM/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de

Original: "They were very intelligent, and they were very intelligent, and they were very intelligent.”
Revised: México/Universidad Nacional Autónoma de México/Universidad Nacional Autónoma de ...”

Original: Pure Sampling :
Revised: Pure Sampling :

Original: The researchers credited these Mages with building Fabian leather armor. Representing an increasing global problem for
Revised: They were cattle called Bolivian Cavalleros; they live in a remote desert uninterrupted by town, and they speak huge, beautiful, paradisiacal Bolivian linguistic thing. They say, 'Lunch, marge.' They don't tell what the lunch is," director

Original: [Table]
Revised: Professor Chuperas Omwell told Sky News. "They've only been talking to scientists, like we're being interviewed by TV reporters. We don't even stick around to be interviewed by TV reporters. Maybe that's how they figured out that they're cosplaying as the Bolivian Cavalleros."

Original:  I NTRODUCTION
Revised:  I NTRODUCTION

Original: On February 14th 2019, OpenAI surprised the scientiﬁc community with an impressively highquality article about Ovid’s Unicorn, written by GPT-2.
Revised: On February 14th 2019, OpenAI surprised the scientiﬁc community with an impressively highquality article about Ovid’s Unicorn, written by GPT-2.

Original: Notably, the top-quality generations obtained from the model rely on randomness in the decoding method, in particular through top-
Revised: Notably, the top-quality generations ob- tained from the model rely on randomness in the decoding method, in particular through top-

Original: k sampling that samples the next word from the top k most probable choices (Fan et al., 2018; Holtzman et al., 2018;
Revised: k sampling that samples the next word from the top k most probable choices (Fan et al., 2018; Holtzman et al., 2018;

Original: Radford et al., 2019), instead of aiming to decode text that maximizes likelihood.
Revised: Radford et al., 2019), instead of aiming to decode text that maximizes likelihood.

Original: In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2-117M, as shown in Figure 1.
Revised: In fact, decoding strategies that optimize for output with high probability, such as beam search, lead to text that is incredibly degenerate, even when using state-of-the-art models such as GPT-2 Large, as shown in Figure 1.

Original: This may seem counter-intuitive, as one would expect that good models would assign higher probability to more human-like, grammatical text.
Revised: This may seem counter-intuitive, as one would expect that good models would assign higher probability to more human-like, grammatical text.

Original: Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.
Revised: Indeed, language models do generally assign high scores to well-formed text, yet the highest scores for longer texts are often generic, repetitive, and awkward.

Revised: Figure 2 exposes how different the distribution of probabilities assigned to beam search decoded text and naturally occurring text really are.

Original: Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling — sampling directly from the probabilities predicted by the model — results in text that is incoherent and almost unrelated to the context.
Revised: Perhaps equally surprising is the right side of Figure 1, which shows that pure sampling — sampling directly from the probabilities predicted by the model — results in text that is incoherent and almost unrelated to the context.

Original: Why is text produced by pure sampling so degenerate?
Revised: Why is text produced by pure sampling so degenerate?

Original: In this work we show that the “unreliable tail” is to blame.
Revised: In this work we show that the “unreliable tail” is to blame.

Original: This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.
Revised: This unreliable tail is composed of tens of thousands of candidate tokens with relatively low probability that are over-represented in the aggregate.

Original: To overcome these shortcomings we introduce Nucleus Sampling ( § 3.1).
Revised: To overcome these issues we introduce Nucleus Sampling ( § 3.1).

Original: The key intuition of Nucleus Sampling is that the vast majority of probability mass at each time step is concentrated in the nucleus , a small subset of the vocabulary that tends to range between one and a thousand candidates.
Revised: The key intuition of Nucleus Sampling is that the vast majority of probability mass at each time step is concentrated in the nucleus , a small subset of the vocabulary that tends to range between one and a thousand candidates.

Original: Instead of relying on a ﬁxed top- k , or using a temperature parameter to control the shape of the distribution without sufﬁciently suppressing the unreliable tail distribution, we propose sampling from the top- p portion of the probability mass, expanding and contracting the candidate pool dynamically.
Revised: Instead of relying on a ﬁxed top- k , or using a temperature parameter to control the shape of the distribution without sufﬁciently suppressing the unreliable tail, we propose sampling from the top- p portion of the probability mass, expanding and contracting the candidate pool dynamically.

Original: In order to compare current methods to Nucleus Sampling, we compare various distributional properties of generated text to the reference distribution, such as the likelihood of veering into repetition and the perplexity of generated text. 
Revised: In order to compare current methods to Nucleus Sampling, we compare various distributional properties of generated text to the reference distribution, such as the likelihood of veering into repetition and the perplexity of generated text. [Figure]

Revised: Timestep

Original: The latter reveals that text generated by maximization or top- k sampling is too probable, indicating a lack of diversity and divergence invocabulary usage from the human distribution.
Revised: Beam Search Text is Less Surprising [Figure] Beam Search ...to provide an overview of the current state-of-the-art in the field of computer vision and machine learning, and to provide an overview of the current state-of-the-art in the field of computer vision and machine learning, and to provide an overview of the current state-of-the-art in the field of computer vision and machine learning, and to provide an overview of the current state-of-the-art in the field of computer vision and machine learning, and... Human ...which grant increased life span and three years warranty.

Revised: The Antec HCG series consists of five models with capacities spanning from 400W to 900W. Here we should note that we have already tested the HCG-620 in a previous review and were quite satisfied

Revised: With its performance.

Revised: In today's review we will rigorously test the Antec HCG-520, which as its model number implies, has 520W capacity and contrary to Antec's strong beliefs in multi-rail PSUs is equipped...

Revised: The latter reveals that text generated by maximization or top- k sampling is too probable, indicating a lack of diversity and divergence in vocabulary usage from the human distribution.

Original: On the other hand, pure sampling produces text that is signiﬁcantly less likely than the gold, corresponding to lower generation quality.
Revised: On the other hand, pure sampling produces text that is signiﬁcantly less likely than the gold, corresponding to lower generation quality.

Original: Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top- k sampling match human statistics.
Revised: Vocabulary usage and Self-BLEU (Zhu et al., 2018) statistics reveal that high values of k are needed to make top- k sampling match human statistics.

Original: Yet, generations based on high values of k are also found to haveincredibly high variance in likelihood, hinting at qualitatively observable incoherency issues.
Revised: Yet, generations based on high values of k often have high variance in likelihood, hinting at qualitatively observable incoherency issues.

Original: Nucleus Sampling can easily match reference perplexity through a proper value of p , without facing theresulting incoherence caused by setting k high enough to match distributional statistics.
Revised: Nucleus Sampling can easily match reference perplexity through tuning the value of p , avoiding the incoherence caused by setting k high enough to match distributional statistics.

Original: Finally, we perform Human Uniﬁed with Statistical Evaluation (HUSE; Hashimoto et al., 2019) to jointly assess the overall quality and diversity of the decoding strategies, which cannot be captured using either human or automatics evaluation alone.
Revised: Finally, we perform Human Uniﬁed with Statistical Evaluation (HUSE; Hashimoto et al., 2019) to jointly assess the overall quality and diversity of the decoding strategies, which cannot be captured using either human or automatic evaluation alone.

Original: The HUSE evaluation demonstrates that Nucleus sampling is the best overall decoding strategy.
Revised: The HUSE evaluation demonstrates that Nucleus Sampling is the best overall decoding strategy.

Original: We include generated examples for qualitative analysis – see Figure 9 for a representative example, and further examples in the appendix.
Revised: We include generated examples for qualitative analysis – see Figure 3 for a representative example, and further examples in the appendix.

Original:  B ACKGROUND 2. T EXT G ENERATION D ECODING S TRATEGIES
Revised:  B ACKGROUND 2. T EXT G ENERATION D ECODING S TRATEGIES

Original: A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2018; Holtzman et al., 2018; Fan et al., 2018).
Revised: A number of recent works have alluded to the disadvantages of generation by maximization, which tend to generate output with high grammaticality but low diversity (Kulikov et al., 2019; Holtzman et al., 2018; Fan et al., 2018).

Original: Generative Adversarial Networks (GANs) have been a prominent research direction (Yu et al., 2017; Xu et al., 2018), but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018; Tevet et al., 2018; Semeniuta et al., 2018).
Revised: Generative Adversarial Networks (GANs) have been a prominent research direction (Yu et al., 2017; Xu et al., 2018), but recent work has shown that when quality and diversity are considered jointly, GAN-generated text fails to outperform generations from language models (Caccia et al., 2018; Tevet et al., 2019; Semeniuta et al., 2018).

Original: Work on neural dialog systems have proposed methods for diverse beam search, using a task-speciﬁc diversity scoring function or constraining beam hypotheses to be sufﬁciently different (Li et al., 2016a; Vijayakumar et al., 2018; Kulikov et al., 2018; Pal et al., 2006).
Revised: Work on neural dialog systems have proposed methods for diverse beam search, using a task-speciﬁc diversity scoring function or constraining beam hypotheses to be sufﬁciently different (Li et al., 2016a; Vijayakumar et al., 2018; Kulikov et al., 2019; Pal et al., 2006).

Original: While such utility functions encourage desirable properties in generations, they do not remove the need to choose an appropriate decoding strategy, and we believe that Nucleus Sampling will have complementary advantages in such approaches. 
Revised: While such utility functions encourage desirable properties in generations, they do not remove the need to choose an appropriate decoding strategy, and we believe that Nucleus Sampling will have complementary advantages in such approaches. Finally, Welleck et al.

Revised: (2020) begin to address the problem of neural text degeneration through an “unlikelihood loss”, which decreases training loss on repeated tokens and thus implicitly reduces gradients on frequent tokens as well.

Revised: Our focus is on exposing neural text degeneration and providing a decoding solution that can be used with arbitrary models, but future work will likely combine training-time and inference-time solutions.

Original: 2. O PEN - ENDED VS DIRECTED GENERATION
Revised: 2. O PEN - ENDED VS DIRECTED GENERATION

Original: Many text generation tasks are deﬁned through (input, output) pairs, such that the output is a constrained transformation of the input.
Revised: Many text generation tasks are deﬁned through (input, output) pairs, such that the output is a constrained transformation of the input.

Original: Example applications include machine translation (Bahdanau et al., 2015), data-to-text generation (Wiseman et al., 2017), and summarization (Nallapati et al., dsfsadfd
Revised: Example applications include machine translation (Bahdanau et al., 2015), data-to-text generation (Wiseman et al., 2017), and summarization (Nallapati et al., 2016). We refer to these tasks as directed generation.

Revised: Typically encoder-decoder architectures are used, often with an attention mechanism (Bahdanau et al., 2015;

Revised: Luong et al., 2015) or using attention-based architectures such as the Transformer (Vaswani et al., 2017).

Original: An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008.
Revised: Generation is usually performed using beam search; since output is tightly scoped by the input, repetition and genericness are not as problematic. Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017) and more recently with exact inference (Stahlberg & Byrne, 2019), a counter-intuitive observation since more comprehensive search helps maximize probability. Open-ended generation , which includes conditional story generation and contextual text continuation (as in Figure 1), has recently become a promising research direction due to signiﬁcant advances in neural language models (Clark et al., 2018; Holtzman et al., 2018; Fan et al., 2018; Peng et al., 2018;

Revised: Radford et al., 2019).

Revised: While the input context restricts the space of acceptable output generations, there is a considerable degree of freedom in what can plausibly come next, unlike in directed generation settings.

Revised: Our work addresses the challenges faced by neural text generation with this increased level of freedom, but we note that some tasks, such as goal-oriented dialog, may fall somewhere in between open-ended and directed generation.

Revised:  L ANGUAGE M ODEL D ECODING

Revised: Given an input text passage as context, the task of open-ended generation is to generate text that forms a coherent continuation from the given context.

Revised: More formally, given a sequence of m tokens x 1 . . .

Revised: x m as context , the task is to generate the next n continuation tokens to obtain the completed sequence x 1 . . .

Revised: x m + n .

Revised: We assume that models compute P ( x 1: m + n ) using the common left-to-right decomposition of the text probability,

Revised: [Equation]

Revised: which is used to generate the generation token-by-token using a particular decoding strategy .

Revised: Maximization-based decoding The most commonly used decoding objective, in particular for directed generation, is maximization-based decoding.

Revised: Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest dsfsadfd

Revised: An unprecedented number of mostly young whales have become stranded on the West Australian coast since 2008.

Original: Poor nutrition has led to a rise in the number of stranded humpback whales on the West Australian coast, veterinary researchers have said.
Revised: Poor nutrition has led to a rise in the number of stranded humpback whales on the West Australian coast, veterinary researchers have said.

Original: Carly Holyoake, from Murdoch
Revised: Carly Holyoake, from Murdoch

Original: University, at the Australian Veterinary Association's annual conference in Perth on Wednesday, said an unprecedented number of mostly young whales had become stranded on the coast since 2008.
Revised: University, at the Australian Veterinary Association's annual conference in Perth on Wednesday, said an unprecedented number of mostly young whales had become stranded on the coast since 2008.

Original: There has been an unprecedented number of calves caught in the nets of whaling stations that operate in WA.
Revised: There has been an unprecedented number of calves caught in the nets of whaling stations that operate in WA.

Original: Pilot whales continue to migrate to feeding grounds to feed their calves.
Revised: Pilot whales continue to migrate to feeding grounds to feed their calves.

Original: They are now vulnerable due to the decline of wild populations; they are restricted to one breeding site each year.
Revised: They are now vulnerable due to the decline of wild populations; they are restricted to one breeding site each year.

Original: Image copyright Yoon Bo Kim
Revised: Image copyright Yoon Bo Kim

Original: But, with sharp decline in wild populations the size of the Petrels are shrinking and dwindling population means there will only be room for a few new fowl.
Revised: But, with sharp decline in wild populations the size of the Petrels are shrinking and dwindling population means there will only be room for a few new fowl.

Original: Pumping Station #3 shut down due to construction damage Find more at: www.abc.net.au/environment/species-worry/ in-the-top-10-killer-whale-catastrophes-in-history.html“Inthetop10killerwhalecatastrophesinhistory:
Revised: Pumping Station #3 shut down due to construction damage Find more at: www.abc.net.au/environment/species-worry/ in-the-top-10-killer-whale-catastrophes-in-history.html“Inthetop10killerwhalecatastrophesinhistory:

Original: 1)1986:Upto12orcasstruckbylightning;manydrowned and many more badly injured.
Revised: 1)1986:Upto12orcasstruckbylightning;manydrowned and many more badly injured.

Original: Last week's intense storms and a series of powerful cyclones have been officially blamed for the deaths of at least nine large fin whales near Whitsundays - the largest loss of any species globally.
Revised: Last week's intense storms and a series of powerful cyclones have been officially blamed for the deaths of at least nine large fin whales near Whitsundays - the largest loss of any species globally.

Original: The fin whales: packed in the belly of one killer whale thrashing madly in fear as another tries to bring it to safety.
Revised: The fin whales: packed in the belly of one killer whale thrashing madly in fear as another tries to bring it to safety.

Original: When the colossal animal breached the waters of Whitsundays, he'd been seen tagged for a decade.
Revised: When the colossal animal breached the waters of Whitsundays, he'd been seen tagged for a decade.

Original: The Australian Food Safety Authority has warned Australia's beaches may be revitalised this year because healthy seabirds and seals have been on the move.
Revised: The Australian Food Safety Authority has warned Australia's beaches may be revitalised this year because healthy seabirds and seals have been on the move.

Original: More than 50, seabirds, sea mammals and seahorses have been swept into the sea by the Holden CS118 and Adelaide Airport CS300 from 2013.
Revised: More than 50, seabirds, sea mammals and seahorses have been swept into the sea by the Holden CS118 and Adelaide Airport CS300 from 2013.

Original: A major white-bat and umidauda migration across Australia is under way in Australia for the first time, with numbers reaching an estimated 50,000.
Revised: A major white-bat and umidauda migration across Australia is under way in Australia for the first time, with numbers reaching an estimated 50,000.

Original: The number of stranded whales has increased by more than 50 per cent in the past year, with the number of stranded whales on the West Australian coast increasing 
Revised: The number of stranded whales has increased by more than 50 per cent in the past year, with the number of stranded whales on the West Australian coast increasing by

Original: [Table]
Revised: [Table]

Revised: Sampling

Original: [Figure]
Revised: [Table]

Original: WebText 
Revised: WebText WebText

Revised: Beam Search, b

Original: [Table]
Revised: [Table]

Original: [Figure][Table] We refer to these tasks as directed generation. Typically encoder-decoder architectures are used, often with an attention mechanisms (Bahdanau et al., 2015;
Revised: [Figure]

Original: Luong et al., 2015) or using attention-based architectures such as the Transformer (Vaswani et al., 2017). Generation is usually performed using beam search; since output is tightly scoped by the input, repetition and genericness are not as problematic. Still, similar issues have been reported when using large beam sizes (Koehn & Knowles, 2017), a counter-intuitive observation since large beam sizes help maximize probability. Open-ended generation , which includes conditional story generation and contextual text continuation (as in Figure 1), has recently become a promising research direction due to signiﬁcant advances in neural language models (Clark et al., 2018; Holtzman et al., 2018;
Revised: Top-

Original: Fan et al., 2018; Peng et al., 2018; Radford et al., 2019). While the input context restricts the space of acceptable output generations, there is a considerable degree of freedom in what can plausibly come next, unlike in directed generation settings. Our work addresses the challenges faced by neural text generation with this increased level of freedom, but we note that some tasks, such as goal-oriented dialog, may fall somewhere in between open-ended and directed generation.
Revised: [Table]

Original: p r o b a b ili t y 0.4 0. I do n’t know .  L ANGUAGE M ODEL D ECODING Given an input text passage as context, the task of open-ended generation is to generate text that forms a coherent continuation from the given context.
Revised: [Figure]

Original: More formally, given a sequence of m tokens x 1 . . . x m as context , the task is to generate the next n continuation tokens to obtain the completed sequence x 1 . . . x m + n . We assume that models compute P ( x 1: m + n ) using the common left-to-right decomposition of the text probability,
Revised: [Table]

Original: [Equation] which is used to generate the generation token-by-token using a particular decoding strategy . Maximization-based decoding The most commonly used decoding objective, in particular for directed generation, is maximization-based decoding. Assuming that the model assigns higher probability to higher quality text, these decoding strategies search for the continuation with the highest likelihood.
Revised: , while likelihood.

Original: Since ﬁnding the optimum argmax sequence from recurrent neural language models or
Revised: Since ﬁnding the optimum argmax sequence from recurrent neural language models or

Original: Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;
Revised: Transformers is not tractable (Chen et al., 2018), common practice is to use beam search (Li et al., 2016b;

Original: Shen et al., 2017; Wiseman et al., 2017).
Revised: Shen et al., 2017; Wiseman et al., 2017).

Original: However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018; Holtzman et al., 2018).
Revised: However, several recent studies on open-ended generation have reported that maximization-based decoding does not lead to high quality text (Fan et al., 2018; Holtzman et al., 2018).

Original: 3. N UCLEUS S AMPLING
Revised: 3. N UCLEUS S AMPLING

Original: We propose a new stochastic decoding method: Nucleus Sampling.
Revised: We propose a new stochastic decoding method: Nucleus Sampling.

Original: The key idea is to use the shape of the probability distribution to determine the set of tokens to be sampled from.
Revised: The key idea is to use the shape of the probability distribution to determine the set of tokens to be sampled from.

Original: Given a distribution
Revised: Given a distribution

Original: P ( x | x 1: i − 1 ) , we deﬁne its top- p vocabulary V ( p ) ⊂
Revised: P ( x | x 1: i − 1 ) , we deﬁne its top- p vocabulary V ( p ) ⊂

Original: V as the smallest set such that
Revised: V as the smallest set such that

Original: [Equation]
Revised: [Equation]

Revised: p r o b a b ili t y 0.4 0.

Revised: I do n’t know .

Original: Let p (cid:48) =
Revised: Let p (cid:48) =

Original: (cid:80) x ∈ V ( p ) P ( x | x 1: i − 1 ) .
Revised: (cid:80) x ∈ V ( p ) P ( x | x 1: i − 1 ) .

Original: The original distribution is re-scaled to a new distribution, from which the next word is sampled:
Revised: The original distribution is re-scaled to a new distribution, from which the next word is sampled:

Original: [Equation]
Revised: [Equation]

Original: In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p .
Revised: In practice this means selecting the highest probability tokens whose cumulative probability mass exceeds the pre-chosen threshold p .

Original: The size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step.
Revised: The size of the sampling set will adjust dynamically based on the shape of the probability distribution at each time step.

Original: For high values of p , it is a small subset of vocabulary that takes up most of the probability mass — the nucleus .
Revised: For high values of p , this is a small subset of vocabulary that takes up vast majority of the probability mass — the nucleus .

Original: 3. T OP - k S AMPLING
Revised: 3. T OP - k S AMPLING

Original: Top- k Sampling has recently become a popular alternative sampling procedure (Fan et al., 2018;
Revised: Top- k sampling has recently become a popular alternative sampling procedure (Fan et al., 2018;

Original: Holtzman et al., 2018; Radford et al., 2019).
Revised: Holtzman et al., 2018; Radford et al., 2019).

Original: Nucleus Sampling and top- k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.
Revised: Nucleus Sampling and top- k both sample from truncated Neural LM distributions, differing only in the strategy of where to truncate.

Original: Choosing where to truncate can be interpreted as determining the generative model’s trustworthy prediction zone.
Revised: Choosing where to truncate can be interpreted as determining the generative model’s trustworthy prediction zone.

Original: At each time step, the top k possible next tokens are sampled from according to their relative probabilities.
Revised: At each time step, the top k possible next tokens are sampled from according to their relative probabilities.

Original: Formally, given a distribution P ( x | x 1: i − 1 ) , we deﬁne its top- k vocabulary V ( k ) ⊂
Revised: Formally, given a distribution P ( x | x 1: i − 1 ) , we deﬁne its top- k vocabulary V ( k ) ⊂

Original: V as the set of size k which maximizes (cid:80) x ∈ V ( k ) P ( x | x 1: i − 1 ) .
Revised: V as the set of size k which maximizes (cid:80) x ∈ V ( k ) P ( x | x 1: i − 1 ) .

Original: Let p (cid:48) =
Revised: Let p (cid:48) =

Original: (cid:80) x ∈ V ( k ) P ( x | x 1: i − 1 ) .
Revised: (cid:80) x ∈ V ( k ) P ( x | x 1: i − 1 ) .

Original: The distribution is then re-scaled as in equation 3, and sampling is performed based on that distribution.
Revised: The distribution is then re-scaled as in equation 3, and sampling is performed based on that distribution.

Original: Note that the scaling factor p (cid:48) will be different at each time-step, in contrast to Nucleus Sampling.
Revised: Note that the scaling factor p (cid:48) can vary wildly at each time-step, in contrast to Nucleus Sampling.

Original: Difﬁculty in choosing a suitable value of k
Revised: Difﬁculty in choosing a suitable value of k

Original: While top- k sampling leads to considerably higher quality text than either beam search or sampling from the full distribution, the use of a constant k is sub-optimal across varying contexts.
Revised: While top- k sampling leads to considerably higher quality text than either beam search or sampling from the full distribution, the use of a constant k is sub-optimal across varying contexts.

Original: As illustrated on the left of Figure 4, in some contexts the head of the next word distribution can be ﬂat across tens or hundreds of reasonable options (e.g. nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the ﬁgure.
Revised: As illustrated on the left of Figure 5, in some contexts the head of the next word distribution can be ﬂat across tens or hundreds of reasonable options (e.g. nouns or verbs in generic contexts), while in other contexts most of the probability mass is concentrated in one or a small number of tokens, as on the right of the ﬁgure.

Original: Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top- k vocabulary will include inappropriate candidates which will have their probability of being sampled increased through the truncation.
Revised: Therefore if k is small, in some contexts there is a risk of generating bland or generic text, while if k is large the top- k vocabulary will include inappropriate candidates which will have their probability of being sampled increased by the renormalization.

Original: Under Nucleus Sampling, the number of candidates considered rises and falls dynamically, corresponding to the changes in the model’s conﬁdence region over the vocabulary which top-
Revised: Under Nucleus Sampling, the number of candidates considered rises and falls dynamically, corresponding to the changes in the model’s conﬁdence region over the vocabulary which top-

Original: k sampling fails to capture for any one choice of k .
Revised: k sampling fails to capture for any one choice of k .

Original: 3. S AMPLING WITH T EMPERATURE
Revised: 3. S AMPLING WITH T EMPERATURE

Original: Another common approach to sampling-based generation is to shape a probability distribution through temperature (Ackley et al., 1985).
Revised: Another common approach to sampling-based generation is to shape a probability distribution through temperature (Ackley et al., 1985).

Original: Temperature sampling has been applied widely to text generation (Ficler & Goldberg, 2017; Fan et al., 2018; Caccia et al., 2018).
Revised: Temperature sampling has been applied widely to text generation (Ficler & Goldberg, 2017; Fan et al., 2018; Caccia et al., 2018).

Original: Given the logits u 1: | V | and temperature t , the softmax is re-estimated as
Revised: Given the logits u 1: | V | and temperature t , the softmax is re-estimated as

Original: [Equation]
Revised: [Equation]

Original: Setting t ∈ [0 , 1) shapes the distributionto be more skewed towards high probability events, which implicitly lowers the mass in the tail distribution.
Revised: Setting t ∈ [0 , 1) skews the distribution towards high probability events, which implicitly lowers the mass in the tail distribution.

Original: Low temperature sampling has also been used to partially alleviate the issues of top- k sampling discussed above, by shaping the distribution before top-
Revised: Low temperature sampling has also been used to partially alleviate the issues of top- k sampling discussed above, by shaping the distribution before top-

Original: k sampling (Radford et al., 2018; Fan et al., 2018).
Revised: k sampling (Radford et al., 2018; Fan et al., 2018).

Original: However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018; Hashimoto et al., 2019).[Table]
Revised: However, recent analysis has shown that, while lowering the temperature improves generation quality, it comes at the cost of decreasing diversity (Caccia et al., 2018; Hashimoto et al., 2019).

Original:  L IKELIHOOD E VALUATION 4. E XPERIMENTAL S ETUP
Revised:  L IKELIHOOD E VALUATION 4. E XPERIMENTAL S ETUP

Original: While many neural network architectures have been proposed for language modeling, including
Revised: While many neural network architectures have been proposed for language modeling, including

Original: LSTMs (Sundermeyer et al., 2012) and convolutional networks (Dauphin et al., 2017), the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018; 2019).
Revised: LSTMs (Sundermeyer et al., 2012) and convolutional networks (Dauphin et al., 2017), the Transformer architecture (Vaswani et al., 2017) has been the most successful in the extremely large-scale training setups in recent literature (Radford et al., 2018; 2019).

Original: In this study we use the Generative
Revised: In this study we use the Generatively

Original: Pre-trained Transformer, version 2 (Radford et al., 2019), which was trained on WebText, a 40GB collection of text scraped from the web.
Revised: Pre-trained Transformer, version 2 (GPT2; Radford et al., 2019), which was trained on WebText, a 40GB collection of text scraped from the web.

Original:  We perform experiments using the Large model (762M parameters).
Revised: 3 We perform experiments using the Large model (762M parameters).

Original: Our analysis is based on generating 5,000 text passages, which end upon reaching an end-of-document token or a maximum length of 200 tokens.
Revised: Our analysis is based on generating 5,000 text passages, which end upon reaching an end-of-document token or a maximum length of 200 tokens.

Original: We consider a conditional generation setting, in which texts are  conditioned on the initial paragraph (restricted to 1-40 tokens) of documents in the validation portion of WebText.
Revised: Texts are generated conditionally, conditioned on the initial paragraph (restricted to 1-40 tokens) of documents in the held-out portion of WebText, except where otherwise mentioned.

Original: 4. P ERPLEXITY
Revised: 4. P ERPLEXITY

Original: Our ﬁrst evaluation is to compute the perplexity of generated text using various decoding strategies, according to the model that is being generated from.
Revised: Our ﬁrst evaluation is to compute the perplexity of generated text using various decoding strategies, according to the model that is being generated from.

Original: We compare these perplexities against that of the gold text (Figure 5).
Revised: We compare these perplexities against that of the gold text (Figure 6).

Original: Importantly, we argue that the optimal generation strategy should produce text which has a perplexity close to that of the gold text: Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to be low-diverstiy and get stuck in repetition loops, as shown in § 5 and illustrated in Figure 3.
Revised: Importantly, we argue that the optimal generation strategy should produce text which has a perplexity close to that of the gold text: Even though the model has the ability to generate text that has lower perplexity (higher probability), such text tends to have low diversity and get stuck in repetition loops, as shown in § 5 and illustrated in Figure 4.

Original: We see that perplexity of text obtained from pure sampling(sampling with t = 1 ) isactually worse than the perplexity of the gold.
Revised: We see that perplexity of text obtained from pure sampling is worse than the perplexity of the gold.

Original: This indicates that the model isactually confusing itself: sampling too many unlikely tokens and creating context that makes it difﬁcult to recover the human distribution of text, ashappens in Figure 1.
Revised: This indicates that the model is confusing itself: sampling too many unlikely tokens and creating context that makes it difﬁcult to recover the human distribution of text, as in Figure 1.

Original: Yet setting the temperature lower creates diversity  repetition issues, as we shall see in § 5.
Revised: Yet, setting the temperature lower creates diversity and repetition issues, as we shall see in § 5.

Original: Interestingly, even with the relatively ﬁne-grained parameter sweep conducted, Nucleus Sampling gets closer perplexity in both the conditional and under conditional case, than any other method, as shown in Table 1. 
Revised: Even with our relatively ﬁne-grained parameter sweep, Nucleus Sampling obtains closest perplexity to human text, as shown in Table 1. [Table]

Original: 4. N ATURAL L ANGUAGE D OES N OT M AXIMIZE P ROBABILITY
Revised: 4. N ATURAL L ANGUAGE D OES N OT M AXIMIZE P ROBABILITY

Original: One might wonder if the issue with maximization  search error is a, i.e., there are higher quality sentences to which the model assigns higher probability than to the decoded ones, beam search has just failed to ﬁnd them.
Revised: One might wonder if the issue with maximization is a search error , i [Table] e., there are higher quality sentences to which the model assigns higher probability than to the decoded ones, beam search has just failed to ﬁnd them.

Original: Yet Figure 5 shows that the per-token probability of natural text is, on average, much lower than text generated by beam search.
Revised: Yet Figures 2 & 6 show that the per-token probability of natural text is, on average, much lower than text generated by beam search.

Original: Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but[Figure] more informative tokens.
Revised: Natural language rarely remains in a high probability zone for multiple consecutive time steps, instead veering into lower-probability but more informative tokens.

Original: Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 3.
Revised: Nor does natural language tend to fall into repetition loops, even though the model tends to assign high probability to this, as seen in Figure 4.

Original: Why is naturally existing human text not the most probable text?
Revised: Why is human-written text not the most probable text?

Original: We conjecture that this is an intrinsic property of human language.
Revised: We conjecture that this is an intrinsic property of human language.

Original: Language models that assign probabilities one word at a time without context for the underlying document will have trouble capturing this effect.
Revised: Language models that assign probabilities one word at a time without a global model of the text will have trouble capturing this effect.

Original: Grice’s Maxims of Communication (Grice, 1975) show that people optimize against stating the obvious.
Revised: Grice’s Maxims of Communication (Grice, 1975) show that people optimize against stating the obvious.

Original: Thus,it is natural that making every word as predictable as possible will be disfavored.
Revised: Thus, making every word as predictable as possible will be disfavored.

Original: This makes solving the problem simply by training larger models or improving neural architectures using standard per-word learning objectives unlikely:  modelsthat areblindly predicting the future will be forced to predict the lowest common denominator, rather than information rich language.
Revised: This makes solving the problem simply by training larger models or improving neural architectures using standard per-word learning objectives unlikely: such models are forced to favor the lowest common denominator, rather than informative language.

Original:  D ISTRIBUTIONAL S TATISTICAL E VALUATION 5. Z IPF D ISTRIBUTION A NALYSIS
Revised:  D ISTRIBUTIONAL S TATISTICAL E VALUATION 5. Z IPF D ISTRIBUTION A NALYSIS

Original: In order to compare generations to the reference text, we begin by analyzing their use of vocabulary.
Revised: In order to compare generations to the reference text, we begin by analyzing their use of vocabulary.

Original: Zipf’s law suggests that there is an exponential relationship between the rank of a word and its frequency in text.
Revised: Zipf’s law suggests that there is an exponential relationship between the rank of a word and its frequency in text.

Original: The Zipﬁan coefﬁcient s can be used to compare the distribution in a given text to a theoretically perfect exponential curve, where s = 

Revised: The Zipﬁan coefﬁcient s can be used to compare the distribution in a given text [Figure] [Figure] a theoretically perfect exponential curve, where s =

Original: (Piantadosi, 2014).
Revised: (Piantadosi, 2014).

Original: Figure 6 shows the vocabulary distributions along with estimated Zipf coefﬁcients for selected parameters of different decoding methods.
Revised: Figure 7 shows the vocabulary distributions along with estimated Zipf coefﬁcients for selected parameters of different decoding methods.

Original: As expected, pure sampling is the closest to the human distribution, followed by
Revised: As expected, pure sampling is the closest to the human distribution, followed by

Original: Nucleus Sampling.
Revised: Nucleus Sampling.

Original: The visualization of the distribution shows that pure sampling slightly overestimates the use of rare words, likely one reason why pure sampling also has higher perplexity than human text.
Revised: The visualization of the distribution shows that pure sampling slightly overestimates the use of rare words, likely one reason why pure sampling also has higher perplexity than human text.

Original: Furthermore, lower temperature sampling avoids sampling these rare words from the tail, which is why it has been used in some recent work (Fan et al., 2018; Radford et al., 2019).
Revised: Furthermore, lower temperature sampling avoids sampling these rare words from the tail, which is why it has been used in some recent work (Fan et al., 2018; Radford et al., 2019).

Original: 5. S ELF -BLEU
Revised: 5. S ELF -BLEU

Original: We follow previous work and compute Self-BLEU (Zhu et al., 2018) as a metric of diversity.
Revised: We follow previous work and compute Self-BLEU (Zhu et al., 2018) as a metric of diversity.

Original: SelfBLEU is calculated by computing the BLEU score of each generated document using all other generations in the evaluation set as references.
Revised: SelfBLEU is calculated by computing the BLEU score of each generated document using all other generations in the evaluation set as references.

Revised: Due to the expense of computing such an operation, we sample 1000 generations, each of which is compared with all 4999 other generations as references .

Original: Thus a lower Self-BLEU score implies higher diversity.
Revised: A lower Self-BLEU score implies higher diversity.

Original: Figure 7 shows that Self-BLEU results largely follow that of the Zipﬁan distribution analysis as a diversity measure.
Revised: Figure 8 shows that Self-BLEU results largely follow that of the Zipﬁan distribution analysis as a diversity measure.

Original: It is worth noting that very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( § 4).
Revised: It is worth noting that very high values of k and t are needed to get close to the reference distribution, though these result in unnaturally high perplexity ( § 4).

Original: 5. R EPETITION
Revised: 5. R EPETITION

Original: One attribute of text quality that we can quantify is repetition.
Revised: One attribute of text quality that we can quantify is repetition.

Original: Figure 8 shows that Nucleus Sampling and top- k sampling have the least repetition for reasonable parameter ranges.
Revised: Figure 9 shows that Nucleus Sampling and top- k sampling have the least repetition for reasonable parameter ranges.

Original: Generations from temperature sampling have more repetition unless very high temperatures are used, which we have shown negatively affects coherence (as measured by high perplexity).
Revised: Generations from temperature sampling have more repetition unless very high temperatures are used which we have shown negatively affects coherence (as measured by high perplexity).

Original: Further, all stochastic methods face repetition issues when their tuning parameters are set too low, which tends to overtruncate , mimicking greedy search.
Revised: Further, all stochastic methods face repetition issues when their tuning parameters are set too low which tends to overtruncate , mimicking greedy search.

Original: Therefore we conclude that only Nucleus Sampling satisﬁes all the distributional criteria for desirable generations.
Revised: Therefore we conclude that only Nucleus Sampling satisﬁes all the distributional criteria for desirable generations

Original:  H UMAN E VALUATION 6. H UMAN U NIFIED WITH S TATISTICAL E VALUATION (HUSE)
Revised:  H UMAN E VALUATION 6. H UMAN U NIFIED WITH S TATISTICAL E VALUATION (HUSE)

Original: Statistical evaluations are unable to measure the coherence of generated text properly.
Revised: Statistical evaluations are unable to measure the coherence of generated text properly.

Original: While the metrics in previous sections gave us vital insights into the different decoding methods we compare, human evaluation is still required to get a full measure of the quality of the generated text.
Revised: While the metrics in previous sections gave us vital insights into the different decoding methods we compare, human evaluation is still required to get a full measure of the quality of the generated text.

Original: However, pure human evaluation does not take into account the diversity of the generated text; therefore we use
Revised: However, pure human evaluation does not take into account the diversity of the generated text; therefore we use

Original: HUSE (Hashimoto et al., 2019) to combine human and statistical evaluation.
Revised: HUSE (Hashimoto et al., 2019) to combine human and statistical evaluation.

Original: HUSE is computed by training a discriminator to distinguish between text drawn from the human and model distributions, [Figure]
Revised: HUSE is computed by training a discriminator to distinguish between text drawn from the human and model distributions, based on only two features: The probability assigned by the language model, and human judgements of typicality of generations.

Original: [Figure] Concretely, a phrase (sequence of words with minimum length 2) must repeat at least three times at the end of  the generation to be considered a repetition. Since all the  methods described used different parameter ranges, we label points with the appropriate parameter value except for t and p .
Revised: Text that is close to the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements. As explored in the previous sections, the current best-performing decoding methods rely on truncation of the probability distribution, which yields a probability of 0 for the vast majority of potential tokens.

Original: Since values of  kgreater than 100 are rarely used in practice andvalues of p are usually in [0 . 9 , 1) , it becomes clear that Nucleus sampling is far closer to  the human distribution in its usual parameter range. Finally, temperatures lower than 0.9 severely effect repetition, making it problematic for use in long-form generation.
Revised: Initial exploration of applying HUSE directly led to top- k and Nucleus Sampling receiving scores of nearly 0 due to truncation, despite humans favoring these methods. As a proxy, when generating the text used to compute HUSE, we interpolate (with mass 0 . 1 ) the original probability distribution with the top- k and Nucleus Sampling distribution, smoothing the truncated distribution.

Original: based on only two features: The probability assigned by the language model, and human judgements of typicality of generations.
Revised: For each decoding algorithm we annotate 200 generations for typicality, with each generation receiving 20 annotations from 20 different annotators.

Original: Text that is close to  the human distribution in terms of quality and diversity should perform well on both likelihood evaluation and human judgements. The results in Table 1 shows that Nucleus Sampling obtains the highest HUSE score, with Top- k sampling performing next best.
Revised: This results in a total of 4000 annotations per a decoding scheme. We use a KNN classiﬁer to compute HUSE, as in the original paper, with k =neighbors, which we found led to the higher accuracy in discrimination. The results in Tableshows that Nucleus Sampling obtains the highest HUSE score, with Top- k sampling performing second best.

Original: 6. Q UALITATIVE ANALYSIS
Revised: 6. Q UALITATIVE A NALYSIS

Original: Figure 9 shows representative example generations.
Revised: Figure 3 shows representative example generations.

Original: Unsurprisingly, Beam Search gets stuck in a repetition loop it cannot escape.
Revised: Unsurprisingly, beam search gets stuck in a repetition loop it cannot escape.

Original: Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word “umidauda”, apparently a species of bird.
Revised: Of the stochastic decoding schemes, the output of full sampling is clearly the hardest to understand, even inventing a new word “umidauda”, apparently a species of bird.

Original: The generation produced by Nucleus Sampling isn’t perfect – the model appears to get get confused that it was talking about whales, and begins talking about birds instead.
Revised: The generation produced by Nucleus Sampling isn’t perfect – the model appears to confuse whales with birds, and begins writing about those instead.

Original: Yet, Top- k Sampling immediately veers off into an unrelated event.
Revised: Yet, top- k sampling immediately veers off into an unrelated event.

Original: When top- k sampling is combine with a temperature of 0.7, as is commonly done (Radford et al., 2019; Fan et al., 2018), the output devolves into repetition.
Revised: When top- k sampling is combined with a temperature of 0.7, as is commonly done (Radford et al., 2019; Fan et al., 2018), the output devolves into repetition, exhibiting the classic issues of low-temperature decoding. More generations are available in Appendix B.

Original:  C ONCLUSION
Revised:  C ONCLUSION

Original: This paper provided a deep analysis into the properties of the most common decoding methods for open-ended language generation.
Revised: This paper provided a deep analysis into the properties of the most common decoding methods for open-ended language generation.

Original: We have shown that likelihood maximizing decoding causes repetition and overly generic language usage, while sampling methods without truncation risk sampling from the low-conﬁdence tail of a model’s predicted distribution.
Revised: We have shown that likelihood maximizing decoding causes repetition and overly generic language usage, while sampling methods without truncation risk sampling from the low-conﬁdence tail of a model’s predicted distribution.

Original: Further, we proposed Nucleus Samplingas a solution that captures the region of conﬁdence of language models effectively.
Revised: Further, we proposed Nucleus Sampling as a solution that captures the region of conﬁdence of language models effectively.

Original: In future work, we wish to dynamically characterize this region of conﬁdence and include a more semantic utility function to guide the decoding process
Revised: In future work, we wish to dynamically characterize this region of conﬁdence and include a more semantic utility function to guide the decoding process. A CKNOWLEDGMENTS

Revised: This research was supported in part by NSF (IIS-1524371), the National Science Foundation Graduate Research Fellowship under Grant No. DGE1256082, DARPA CwC through ARO (W911NF151- 0543), DARPA MCS program through NIWC Paciﬁc (N66001-19-2-4031), the South African

Revised: Centre for Artiﬁcial Intelligence Research, and the Allen Institute for AI.

Original: A E XAMPLE GENERATIONS
Revised: B E XAMPLE GENERATIONS

Original: We include another set of examples for further qualitative comparison.
Revised: We include a set of examples for further qualitative comparison.

Original: [Table]
Revised: [Table]

Original: When will you start sending this stuff for register until someone else orders?
Revised: When will you start sending this stuff for register until someone else orders?

Original: If anyone wants to order, we would like to keep the next batch of binders as being a vaiible sample.
Revised: If anyone wants to order, we would like to keep the next batch of binders as being a vaiible sample.

Original: It ASKED for the logistics and cost of shipping.
Revised: It ASKED for the logistics and cost of shipping.

Original: I wish for a refund if actions like this are to set us back another beat.
Revised: I wish for a refund if actions like this are to set us back another beat.

Original: Are there any instructions for understanding the CDI
Revised: Are there any instructions for understanding the CDI

Original: CAL 0519-019?
Revised: CAL [Table]

Original: If there are, I'd be very keen to find them.
Revised: If there are, I'd be very keen to find them.

Original: What's the best way to read this one?
Revised: What's the best way to read this one?

Original: The blog post from the authors discusses the differences between how the marketing mix plays out in terms of visibility, trustworthiness, and engagement, among other things.
Revised: The blog post from the authors discusses the differences between how the marketing mix plays out in terms of visibility, trustworthiness, and engagement, among other things.

Original: In my experience, this difference isn't necessarily obvious; it's hard to explain to a layperson, but in all honesty it's probably the most important point to consider when reviewing social media.
Revised: In my experience, this difference isn't necessarily obvious; it's hard to explain to a layperson, but in all honesty it's probably the most important point to consider when reviewing social media.

Original: 8 questions
Revised: 8 questions

Original: [Table]

Original: [Table]

Original: I have a small pad for the calf and on the off

Original: [Table]
Revised: [Table]

Original: it all breaks i would like
Revised: 28th February

Original: [Table]
Revised: [Table]

Original: know that how do you square up the circumference of [Table] something as
Revised: : Jaybo

Original: [Table]
Revised: [Table]

Original:  as a new t shirts [Table] at
Revised: : Durability I have a small pad for the calf and on the off chance it all breaks i would like to know that how do you square up the circumference of the pad with something as comfortable as a new t shirts.

Original: [Table]
Revised: Thanks.

Original: Kiran

Original: [Table] 28th February [Table]
Revised: Has the compression system at all moved???? [Table]

Revised: Asked By: Kiran R28th February 2016 Product: Jaybo Comfort: Durability

Original: How can I change the wallpaper shipping
Revised: How can I change the wallpaper for my computer screen?

Original: We do love our in 
Revised: If you have the latest (current version) of your operating system, then you change the background for your How wallpaper change will apply. How can I change Where can I purchase my prototypes and printed books in the USA?

Revised: Currently we have a sign for purchase available in the guide.

Original:  We 
Revised: Since we are based in the States we will be shipping to you by FedEx. Do you make books in America? We do! We love our folks in the US.

Original: 't had a  to make a documentary so  love to share our  for our 
Revised: We haven't had a chance to make a documentary so we love to share our love for our culture.

Original: How does the book  pay for the  art?
Revised: How does the book publisher pay for the book and art?

Original: Why are your caustics superior?
Revised: Why are your caustics superior?

Original: Purest Grades Available.
Revised: Purest Grades Available.

Original: All US Manufactured.
Revised: All US Manufactured.

Original: No Asian Imports
Revised: No Asian Imports.

Original: Quality Control on all incoming and outgoing shipments.
Revised: Quality Control on all incoming and outgoing shipments.

Original: Most suppliers only supply tech grade product.
Revised: Most suppliers only supply tech grade product.

Original: What is difference between Food and Tech Grade Lye?
Revised: What is difference between Food and Tech Grade Lye?

Original: Food grade has a lower parts per million of heavy metal impurities.
Revised: Food grade has a lower parts per million of heavy metal impurities.

Original: 80% of our soap makers prefer our food grade product. 
Revised: 80% of our soap makers prefer our food grade product. [Table] [Figure] [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: [Table]

Revised: [Figure]

Revised: dsfsadfd

Revised: University of Wisconsin — Madison

Revised: In late 1998, a UW-Madison group led by James Thomson was the first to isolate and culture human embryonic stem cells, master undifferentiated cells that arise at the earliest stages of development and are capable of becoming any of the 220 types of cells and tissues in the human body.

Original: {+Thomas Jefferson, 1777—
+}
Revised: {+Thomas Jefferson, 1777—
+}

Revised: Who Is Mike Barnacle?

Revised: Well, at the start, this was clearly a project designed to help people get the gist of classic art in its myriad ways.

Revised: Now we find ourselves with an interesting set of recordings of 200 BC.

Revised: Who are these guys?

Revised: This one, apparently, are the descendants of Greek historian Euclid.

Revised: He famously analyzed straight lines so we know those are straight.

Revised: Colorado State University Denver Claremont McKenna College

Revised: College New

Revised: Institute of Technology

Revised: School

Revised: Medicine Wisconsin School

Revised: Engineering

Revised: University of Wisconsin —

Revised: University

Revised: University

Revised: — Racine University

Revised: — Red Wing

Revised: University of

Revised: Point University

Revised: Point

Revised: Jonathan Hedberg Department of Comparative and Applied

Revised: University

Revised: Michael Amante Department of Economics

Revised: Mental Health 4240 New Florence Rd, Henderson, WI 54711 (262)-335MentalHealth@uiowa.eduUniversityofWisconsinMadison (at UW Campus)

Revised: University of Wisconsin — Madison University of Wisconsin — Milwaukee

Revised: University of Wisconsin — Oshkosh

Revised: Sampling,

Revised: [Figure]

Revised: =0.

Revised: WebText

Revised: WebText

Revised: [Figure]

Revised: Search,

Revised: [Figure]

Revised: =

Revised: [Figure]

Revised: Sampling

Revised: [Figure]