Sparse Text Generation (2020)

Abstract: Current state-of-the-art text generators build on powerful language models such as GPT-2, achieving impressive performance. However, to avoid degenerate text, they require sampling from a modified softmax, via temperature parameters or ad-hoc truncation techniques, as in top- k or nucleus sampling. This creates a mismatch between training and testing conditions. In this paper, we use the recently introduced entmax transformation to train and sample from a natively sparse language model, avoiding this mismatch. The result is a text generator with favorable performance in terms of fluency and consistency, fewer repetitions, and n-gram diversity closer to human text. In order to evaluate our model, we propose three new metrics for comparing sparse or truncated distributions: \epsilon -perplexity, sparsemax score, and Jensen-Shannon divergence. Human-evaluated experiments in story completion and dialogue generation show that entmax sampling leads to more engaging and coherent stories and conversations.

## introduction the last few years have seen a tremendous progress in neural language modeling, with a strong impact on key nlp tasks such as machine translation @xcite @xcite , speech recognition @xcite , summarization @xcite , and open-ended text generation @xcite . benefiting from large amounts of data, models such as transformer-xl @xcite and gpt-2 @xcite have achieved impressive perplexity scores on language modeling. however, the generated text is still often repetitive and incoherent ( a downside of current approaches is the mismatch between training and testing conditions: context:"are they ready to go live?" zuko asked. "yup, in just a minute" tomi replied as he leaned back in his reclining chair. this was a moment that the entire galaxy would see. after 10 years... greedy: ... of study, the scientists had finally come to the conclusion that the universe was not a place for living. the scientists had been studying the universe for over a century. they had been studying the universe for over a century. top-k: ... of studying this phenomenon, the team had finally come up with a plan to study what the aliens were doing. the team would be looking for a way to create a space station. "and what would you call it?" "what would you call the space station?" nucleus: ... of diligent testing, and some test run after that, the results were well received. "they're waiting for you, because there's no way they would." stowedly at a local wedding place, the two archaeologists spoke together. entmax: ... of hibernation, tomi was back. these last days had been a significant step forward in his mission. this time, tomi was not alone. all the empires had aligned together and the world's leadership began to openly support his mission. a second problem is that it is hard to compare different truncation strategies-for example, we cannot easily evaluate how the resulting truncated distributions behave as language models, since the most widely used metric for language modelingperplexity-cannot handle sparse distributions: if a model assigns zero probability to a single reference word, it gets infinite perplexity for the entire sample. for this reason, previous works generate from a truncated softmax, but report the perplexity of the full softmax distribution @xcite . others use the latter to compare perplexity on the generated text with that on human text @xcite 2), or resort to distributional statistics @xcite . in this paper, we propose a new approachentmax sampling ( §3)-that eliminates the mismatch between training and test conditions. key to our approach is the recently proposed entmax transformation @xcite . entmax transforms a vector of scores into a sparse probability distribution, preventing implausible words from receiving any probability mass. moreover, it does so natively: it comes with a well-defined loss function that allows it to learn its sparsity automatically from the data, during training. this results in a new stochastic text generator where the number of possible word types varies with the context (like nucleus sampling), but that generates by sampling directly from its output distribution (like softmax), and where the sparsity of this distribution is present during training (unlike any existing method). as a second contribution, we propose three new metrics to support the evaluation of sparse language models ( §4): -perplexity, sparsemax score, and jensen-shannon divergence. we show that these metrics are well supported theoretically and can be used to compare our method with various truncation and temperature techniques. experiments in language modeling, story completion, and dialogue generation ( §5) show that entmax sampling generates more diverse text and fewer repetitions than nucleus and top-k sampling. foot_0 diversity-promoting models. in addition to new decoding methods, models that aim to increase word diversity and diminish repetition have also been introduced. @xcite al. (2018) (2019) via their α-entmax transformation, which was applied to sequence-to-sequence models for morphological inflection and machine translation. in contrast to our work, they performed deterministic decoding with beam search, and they did not consider open-ended generation. evaluation metrics. the most common metrics to evaluate text generation models are perplexity @xcite and bleu @xcite . for open-ended generation, @xcite observed that "no single metric is comprehensive enough". other evaluations include corpus n-gram overlap @xcite , and the fréchet distance @xcite . these approaches are aimed at the (harder) problem of evaluating the quality of generated text. by contrast, our paper proposes new metrics for evaluating language models in the task of predicting the next word conditioned on ground truth context (like perplexity does), but supporting sparse probability distributions (which perplexity does not). ## language modeling language models assign probability to word sequences @xmath0, x 1 , . . . , x t , stop , where each x t is in a vocabulary v, and @xmath1. this probability can be written as p θ (x) = t +1 @xmath2. we would like the model θ to assign high probability to real sentences, i.e., each distribution p θ (• | @xmath3. given a set s of training sentences, the usual strategy for learning the language model parameters θ is to minimize the negative log-likelihood: the standard choice to model p θ (•|@xmath4. 1 is to compute a score vector z t by conditioning on the context @xmath5, and then applying a softmax transformation, p θ (•|@xmath6, where at decoding time, the language model generates sentences one word at a time, by sampling from the learned probability distribution. however, softmax yields a dense distribution, i.e., some probability mass (even if small) is assigned to all the words in the vocabulary. @xcite have shown that, if we sample from this distribution directly, the resulting text becomes degenerate, with common incoherences arising due to the unreliability of the tail of the distribution. this motivated a line of work proposing "ad-hoc" modifications to the softmax distribution, to reduce the effect of the tail. two of the most successful techniques, top-k and nucleus sampling @xcite , do so by truncating and renormalizing the distribution p θ (•|@xmath7. note that these techniques are applied only at decoding time-during training the original softmax distribution is left untouched, being used as part of the optimization of the cross-entropy loss. our alternative to these ad-hoc modifications builds on learnable sparse transformations, as we shall see in §3. these transformations can produce sparse, zero-tailed probability distributions, learning the amount of sparsity from data. therefore, sampling from these distributions directly is a natural way to prevent degenerate text. ## entmax sampling key to our method is the recently proposed αentmax family of transformations 2 @xcite , parametrized by a scalar parameter α ≥ 1: α-entmax(z t ) := argmax with α = 1 and α = 2, we recover the shannon and gini entropies, respectively. 3 when α → ∞, h α (p) → 0. thus, 1-entmax, 2-entmax, and ∞-entmax recover softmax, sparsemax, and argmax, respectively. blondel et al. (2019) have shown that, for α > 1, entmax is able to output sparse probability distributions, where some words get exactly zero probability, whereas softmax (α = 1) does not have this capability. when α = 1, we still recover the negative loglikelihood, ℓ α (z t , x) = -log p θ (x), and, when α = 2, this corresponds to the sparsemax loss @xcite , to be revisited in §4. entmax losses belong to the wider class of fenchel-young losses @xcite and, consequently, are convex on z and differentiable (with gradient ∇ z ℓ α (z, x) = -e x + p θ ). for α > 1, they have a separation margin: the loss is zero iff z @xmath8, in which case p θ = e x , i.e., the model puts all its probability mass in the correct word. this allows the model to be adaptive to the degree of uncertainty present: in some cases there are few plausible words, so most words should have probability zero, while in other cases a higher number of words are plausible and should be given probability mass. entmax sampling. at test time, we simply sample from the categorical distribution obtained by applying the entmax transformation to the scores z t given by the model: note that, in contrast to previously proposed methods such as top-k sampling and nucleus sampling @xcite , we sample directly from the learned sparse probability distribution over the words, without any calibration or ad-hoc modification. as in nucleus sampling and in opposition to top-k sampling, entmax sampling considers a varying number of tokens depending on the context. moreover, as we show in ## evaluation metrics language models are commonly evaluated by computing their perplexity (ppl) on held-out data. perplexity assesses the ability of a language model to predict the next word given the context: however, its computation involves the logarithm of a probability. this poses a problem when we are using sparse or truncated probability distributions, since lim @xmath9. usually, authors report the values for perplexity computed on the original probability distribution, before truncation. however, this metric does not allow different sparse decoding strategies to be compared. 4 as an alternative, we propose three different metrics (to better understand these metrics, comparative plots are shown in fig. 2 , app. e). ## experiments we compare the different methods in three nlp tasks: language modeling ( §5.1), story completion ( §5.2), and dialogue generation ( §5.3). in language modeling, we evaluate the model's fluency, while in story completion we also evaluate if the methods generate coherent and "interesting" text. in dialogue generation, we evaluate the methods' performance in an interactive task. ## conclusions we proposed entmax sampling as a new strategy for generating text from a sparse probability distribution. it provides three main advantages: (i) it offers a natural way of sampling directly from the output probability distribution; (ii) the distribution sparsity is modeled during training, avoiding a sparsity mismatch between training and run time; (iii) when sampling with entmax, the number of words to be considered varies with the context, as in nucleus sampling and in contrast to top-k sampling. additionally, we proposed new metrics for evaluating language models that produce sparse and truncated probability distributions: -perplexity, sparsemax score, and jensen-shannon divergence. experiments show that entmax sampling leads to higher n-gram diversity, fewer repetitions, and sim-ilar or improved results in automatic metrics. human evaluation confirms that entmax outperforms greedy decoding, top-k, and nucleus sampling in coherence/consistency and engagement, and is similar or better in terms of fluency.