M ix CE : Training Autoregressive Language Models by Mixing Forward and Reverse Cross-Entropies (2023)

Abstract: Autoregressive language models are trained by minimizing the cross-entropy of the model distribution Q relative to the data distribution P – that is, minimizing the forward cross-entropy, which is equivalent to maximum likelihood estimation (MLE). We have observed that models trained in this way may “over-generalize”, in the sense that they produce non-human-like text. Moreover, we believe that reverse cross-entropy, i.e., the cross-entropy of P relative to Q, is a better reflection of how a human would evaluate text generated by a model. Hence, we propose learning with MixCE, an objective that mixes the forward and reverse cross-entropies. We evaluate models trained with this objective on synthetic data settings (where P is known) and real data, and show that the resulting models yield better generated text without complex decoding strategies.

## introduction rapid advances in pre-trained large-scale autoregressive language models (lms) have dramatically improved the performance of a variety of tasks @xcite @xcite . however, these systems still struggle in many openended generation settings, where they are asked to produce a long text following a short prompt. in these cases, we seek systems that generate sensical, coherent, fluent, and engaging, or in short, human-like text @xcite . different decoding strategies to generate such text from pretrained lms suffer from different degeneration problems. unbiased sampling 1 usually results in incoherent and nonsensical text, while greedy and beam searches often get stuck in repetition loops @xcite . these observations suggest that the learned lm distribution q θ still differs substantially from the human lm distribution p . a possible reason is that the autoregressive modeling of q θ gives a non-zero probability to every possible sequence of tokens, while many sequences are impossible under p . nevertheless, we still hope that q θ (x) is as small as possible when p (x) = 0. to this end, maximum likelihood estimation (mle), i.e., minimizing the cross-entropy (ce) -e x∼p [log q θ (x)], is the most widely used objective to train q θ (x) using sequences sampled from p . in an idealized setting, with unlimited training data and model capacity, as well as a perfect optimizer, fitting q θ with mle will learn a distribution as close to p as we like. however, in practice, we only have finite and noisy data. we argue that the mle objective only weakly penalizes generations x from q θ that are "bad", in the sense that p (x) = 0. when q θ puts a small amount of probability mass onto p (x) = 0 space, mle cannot sufficiently discourage this behavior (see figure 3 in appendix c). moreover, minimizing forward ce, -e x∼p [log q θ (x)], is equivalent to minimizing the forward kl divergence between p and q θ , i.e., kl(p ||q θ ) = e x∼p [log p (x)/q θ (x)]. forward kl has a zeroavoiding property -avoiding q θ (x) = 0 when p (x) ̸ = 0 @xcite . therefore, if there is noise in the data, q θ will try to cover the noise as well, which leads the model to over generalize, in the sense of putting non-trivial probability mass over p (x) = 0 generations @xcite @xcite . as a result, we observe samples from the model deviating from human-like text. a common strategy is to modify the decoding method, e.g., top-k, top-p, typical, contrastive @xcite @xcite samplings, to tailor the model distribution q θ in a post-hoc manner to avoid unwanted generations. in contrast, our approach differs: how can we obtain a better q θ to obviate the need for these sampling strategies? we propose a novel training objective for autoregressive lms -mixce that mixes the forward and reverse cross-entropies: @xcite ]. mixce can be understood in two ways. first, we want model generations to be high-quality as well as diverse. reverse cross-entropy reflects how we conduct human evaluations, sampling from the model q θ and evaluating it by the human p , where the focus is text quality. forward cross-entropy emphasizes the diversity of model generations @xcite . second, mixce works similarly to a mixture of the forward and reverse kl divergences. the reverse kl divergence (kl(q θ ||p )) is zero-forcing -forcing q θ (x) = 0 when p (x) = 0 -and thus more strongly penalizes generating non-human-like samples compared to mle. overall, mixce combines two complementary driving forces to better fit q θ to p (figure 1 ). we elaborate on these interpretations in § 3.1. unfortunately, optimizing reverse cross-entropy is intractable because we do not know p . hence, we propose an approximation of the reverse crossentropy (see § 3.2), which ends up being a selfreinforced loss function that encourages the model to produce generations in which it is already con-fident. this loss function has the same computational complexity as forward cross-entropy, making mixce easy to implement and as fast as mle. we demonstrate the effectiveness of mixce in both a synthetic setting, where the "human" distribution p is known, as well as a real setting. for the synthetic case, we evaluate six learning objectives: mixce, mixce * (mixce without approximation), forward kl (=mle), reverse kl, the mixture of two kl divergences, and jensen-shannon (js) divergence. we show that mixce * works slightly worse than the mixture of kls while outperforming other objectives, and mixce works worse than mixce * but generally outperforms mle. in real settings, we finetune gpt-2 @xcite of different sizes on three english text domains using mixce or mle. our results show that, compared to mle, unbiased sampling from mixce-finetuned models produces text that has diversity @xcite closer to that of human text, has higher coherence @xcite , has higher mauve @xcite , and is preferred by humans. when using top-p sampling @xcite and carefully tuning p, generations from mle-finetuned models are similar to those generated from mixce-finetuned models. nonetheless, mixce models have tuned p values closer to 1, implying a less noisy model distribution. in addition, we modify the original mauve to make it more robust to spurious features (e.g., text length), under which mixce still improves over mle when using unbiased sampling. ## conclusion we propose a novel training objective, mixce, for autoregressive language modeling. mixce combines forward and reverse cross-entropies, which can be viewed as combining two complementary driving forces for better fitting the model distribution to the data distribution. we demonstrate the superiority of mixce over mle in both synthetic and real settings via both automatic and human evaluations. in the future, mixce can be potentially used for pretraining language models.