A Contrastive Framework for Neural Text Generation
Abstract
Text generation is of great importance to many natural language processing applications.
However, maximization-based decoding methods (e.g., beam search) of neural language models often lead to degenerate solutions—the generated text is unnatural and contains undesirable repetitions.
Existing approaches introduce stochasticity via sampling or modify training objectives to decrease the probabilities of certain tokens (e.g., unlikelihood training).
However, they often lead to solutions that lack coherence.
In this work, we show that an underlying reason for model degeneration is the anisotropic distribution of token representations.
We present a contrastive solution: (i) SimCTG , a contrastive training objective to calibrate the model’s representation space, and (ii) a decoding method— contrastive search —to encourage diversity while maintaining coherence in the generated text.
Extensive experiments and analyses on three benchmarks from two languages demonstrate that our proposed approach significantly outperforms current state-of-the-art text generation methods as evaluated by both human and automatic metrics.
Introduction
Open-ended neural text generation with Transformer [52] is an indispensable component in various natural language applications, such as story generation [11, 43], contextual text completion [36], and dialogue systems [48].
However, the conventional approach of training a language model with maximum likelihood estimation (MLE) and decoding the most likely sequence is often not sufficient [14, 54].
Specifically, this modelling formulation often leads to the problem of degeneration , i e., the generated texts from the language model tend to be dull and contain undesirable repetitions at different levels (e.g., token-, phrase-, and sentence-level) [8].
To alleviate this problem, previous solutions modify the decoding strategy by sampling from less likely vocabularies [11, 14].
While reducing the generated repetition, these sampling methods introduce another critical problem ( semantic inconsistency )—the sampled text tends to diverge from or even contradict to the original semantics defined by the human-written prefix [3].
Another approach addresses the degeneration problem by modifying the model’s output vocabulary distribution with unlikelihood training [54].
In this work, we argue that the degeneration of neural language models stems from the anisotropic distribution of token representations, i.e., their representations reside in a narrow subset of the entire space [10, 9, 44].
In Figure 1(a), we showcase a cosine similarity matrix of token representations (taken from the output layer of the Transformer) produced by GPT-2.
We see that the cosine similarities between tokens within a sentence are over 0.95 , meaning that these representations are close to each other.
Such high similarity is undesirable as it can naturally cause the model to generate repetitive tokens at different steps.
In an ideal setting, the token representations should follow an isotropic distribution, i.e., the token similarity matrix should be sparse and the representations of distinct tokens should be discriminative as shown in Figure 1(b).
Moreover, during the decoding process, the sparseness of the token similarity matrix of the generated text should be preserved to avoid model degeneration.
Based on the above motivations, we present SimCTG (a sim ple c ontrastive framework for neural t ext g eneration) that encourages the model to learn discriminative and isotropic token representations.
We also present a novel decoding strategy to complement SimCTG, contrastive search .
The key intuitions behind contrastive search are: (i) at each decoding step, the output should be selected from the set of most probable candidates predicted by the model to better maintain the semantic coherence between the generated text and the human-written prefix, and (ii) the sparseness of the token similarity matrix of the generated text should be preserved to avoid degeneration.
We conduct comprehensive experiments on three widely used benchmarks.
We show that our approach is generalizable to different tasks and different languages (§4 and §5) as well as different model sizes (§4.3 and Appendix D).
Specifically, the experimental results verify that SimCTG improves the intrinsic qualities of the language model, as evaluated by perplexity and token prediction accuracy (§4.2 and Appendix D).
Moreover, we demonstrate that the proposed contrastive search significantly outperforms previous state-of-the-art decoding methods in both human and automatic evaluations (§ and §5).
Furthermore, we provide in-depth analyses to get better insights on the inner-workings of our proposed approach (§6).
Background 2.1 Language Modelling
The goal of language modelling is to learn a probability distribution p θ ( x ) over a variable-length text sequence x = { x 1 , ..., x | x | } , where θ denotes model parameters.
Typically, the maximum likelihood estimation (MLE) objective is used to train the language model which is defined as
[Equation]
However, as observed in many recent studies [10, 9, 44], training with likelihood maximization objective often yields an anisotropic distribution of model representations (especially for Transformerbased models) that undermines the model’s capacity.
2.2 Open-ended Text Generation
In this work, we focus on studying the task of open-ended text generation due to its generality in various applications, such as story generation [11, 43], contextual text completion [36], poetry generation [23], and dialogue systems [48].
Formally, conditioned on a human-written prefix (i.e., context) x , the task is to decode a continuation ˆ x from the language model and the resulting text is { x 1 , .., x | x | , ˆ x | x | +1 , ..., ˆ x | x | + | ˆ x | } .
Typically, there are two classes of methods used for decoding, which are (1) deterministic methods and (2) stochastic methods.
Deteriminstic Methods.
Two widely used deterministic approaches are greedy and beam search which aim to select the text continuation with highest probability based on the model’s probability distribution p θ .
However, solely maximizing the output probability often leads to dullness [22] and degeneration [11, 14] in the generated text.
Stochastic Methods.
To remedy the issues of deterministic decoding, several approaches have been proposed to sample from p θ .
To avoid sampling from the unreliable tail of distribution, Fan et al.
[11] proposed top-
k sampling which draws sample from the vocabulary subset V ( k ) that maximizes (cid:80) v ∈ V ( k ) p θ ( v | x ) .
Here, | V ( k ) | =
k and x is the prefix context.
Differently, the current state-of-the-art nucleus sampling [14] draws sample from the smallest vocabulary subset U with total probability mass above a threshold p ∈ [0 , 1] ; i.e., U is the smallest vocabulary subset such that (cid:80) v ∈ U p θ ( v | x ) ≥ p .
While the sampling approaches help to alleviate model degeneration, the intrinsic stochasticity in these methods could cause the semantic meaning of the sampled text to diverge from or even contradict to the human-written prefix [3].
Methodology
In this section, we first present how to apply contrastive learning to calibrate the representation space of the language model.
Then, we introduce our proposed contrastive search decoding algorithm.
3.1 Contrastive Training
Our goal is to encourage the language model to learn discriminative and isotropic token representations.
To this end, we introduce a contrastive objective L CL into the training of the language model.
Specifically, given a variable-length sequence x = { x 1 , ..., x | x | } , the L CL is defined as
[Equation]
where ρ ∈ [ − 1 , 1] is a pre-defined margin and h x i is the representation of token x i produced by the model.
The similarity function s computes the cosine similarity between token representations as
[Equation]
Intuitively, by training with L CL , the model learns to pull away the distances between representations of distinct tokens.
Therefore, a discriminative and isotropic model representation space can be obtained.
The overall training objective L SimCTG is then defined as
[Equation]
where the maximum likelihood estimation (MLE) objective L MLE is described in Eq.
Note that, when the margin ρ in L CL equals to 0 , the L SimCTG degenerates to the vanilla MLE objective L MLE .
3.2 Contrastive Search
We propose a novel decoding method, contrastive search .
At each decoding step, the key ideas of contrastive search are (i) the generated output should be selected from the set of most probable candidates predicted by the model; and (ii) the generated output should be discriminative enough with respect to the previous context.
In this way, the generated text can (i) better maintain the semantic coherence with respect to the prefix while (ii) avoiding model degeneration.
Formally, given the previous context x <t , at time step t , the selection of the output x t follows
[Equation] [Table]
where V ( k ) is the set of top- k predictions from the model’s probability distribution p θ ( ·| x <t ) and k is typically set as 3 ∼ 10.
In Eq.
(5), the first term, model confidence , is the probability of candidate v predicted by the model.
The second term, degeneration penalty , measures how discriminative of candidate v with respect to the previous context x <t and s is defined in Eq.
Specifically, it is defined as the maximum cosine similarity between the representation of v and that of all tokens in x <t .
Here, the candidate representation h v is computed by the model given the concatenation of x <t and v .
Intuitively, a larger degeneration penalty of v means it is more similar to the context, therefore more likely leading to model degeneration.
The hyperparameter α ∈ [0 , 1] regulates the importance of these two components.
When α = 0 , contrastive search degenerates to the greedy search method.
Document Generation
We first evaluate our approach on the task of open-ended document generation.
Model and Baselines.
Our proposed approach is architecture-agnostic and can be applied to any generation model.
In this work, we evaluate our method on the representative GPT-2 model [36].
Specifically, we fine-tune GPT-2 on the evaluated benchmark (detailed below) with the proposed objective L SimCTG (Eq.
and generate the text continuation with different decoding methods.
We perform experiments using the base model (117M parameters) which consists of 12 Transformer layers [52] with 12 attention heads.
3 We compare our approach with two strong baselines: (1) GPTfine-tuned with the standard MLE objective (Eq.
(1)); and (2) GPT-2 fine-tuned with unlikelihood objective [54].
4 Our implementation is based on the Huggingface Library [56].
Evaluation Benchmark.
We conduct experiments on the Wikitext-103 dataset [33] which contains a large collection of Wikipedia articles with over 100 million words and 260 thousands unique tokens.
Wikitext-103 is a document-level dataset and has been widely used for the evaluation of large-scale language modelling [6, 16, 58].
Training.
For our SimCTG and the MLE baseline, we fine-tune the models on Wikitext-103 for 40k training steps.
For the unlikelihood baseline, following Welleck et al.
[54], we first fine-tune the model with the token-level unlikelihood objective for 38.5k steps and then with the sequence-level unlikelihood objective for 1.5k steps.
Therefore, the overall training steps of all compared methods are the same.
The batch size is set as 128 and the training samples are truncated to a maximum length of 256.
We optimize the model with Adam optimizer [17] and a learning rate of 2e-5.
Decoding.
We evaluate the models by producing text continuations given the prefixes from the test set.
In the experiments, the lengths of the prefix and the generated continuation are set as 32 and 128, respectively.
We test different models with various decoding methods.
For deterministic method, we use greedy search and beam search with a beam size of 10 .
For stochastic method, we use the current state-of-the-art nucleus sampling [14] with p = 0 .
For the proposed contrastive search, the k and α in Eq.
(5) are set as 8 and 0 .
The hyperparameters of different methods are selected based on their optimal MAUVE (detailed in §4.1.2) performance on the validation set.
4.1 Evaluation Metrics
We perform evaluation from two aspects: (1) language modelling quality which measures the intrinsic quality of the model; and (2) generation quality which measures the quality of the generated text.
4.1.1 Language Modelling Quality
Following Welleck et al.
[54], we report the results of the model on the metrics below.
Perplexity.
The model perplexity ( ppl ) on the test set of Wikitext-103.
Prediction Accuracy.
It is defined as: acc =
[Equation]
where D is the Wikitext [Table] results on Wikitext-103 test set. “Unlike.” denotes the model trained with unlikelihood objective.
↑ means higher is better and ↓ means lower is better.
Prediction Repetition.
The fraction of next-token (top-1) predictions that occur in the prefix which is defined as: rep =
[Equation]
In addition, the next token repetitions that do not equal to the ground truth token: wrep = 1 (cid:80) x ∈D | x | (cid:80) x ∈D (cid:80) | x | t =1 1 [arg max p θ ( x | x <t ) ∈
x
<t ∧ ̸ = x t ] is also reported.
4.1.2 Generation Quality
Generation Repetition.
This metric measures the sequence-level repetition as the portion of duplicate n -grams in the generated text [54].
For a generated text continuation ˆ x , the repetion at n -gram level is defined as: rep-n = 100 × (1 . 0 − | unique n-grams (ˆ x ) | | total n-grams (ˆ x ) | ) .
Diversity.
This metric takes into account the generation repetition at different n -gram levels and it is defined as: diversity =
(cid:81) 4 n =2 (1 . 0 − rep-n 100 ) .
It can be deemed as an overall assessment of model degeneration.
A lower diversity means a more severe degeneration of the model.
MAUVE [34] is a metric that measures the token distribution closeness between the generated text and human-written text.
A higher MAUVE score means the model generates more human-like texts.
Semantic Coherence.
To automatically measure the semantic coherence (i.e., consistency) between the prefix and the generated text, we employ the advanced sentence embedding method, SimCSE [13].
Specifically, given the prefix x and the generated text ˆ x , the coherence score is defined as: coherence = v ⊤ x v ˆ x / ( ∥ v x ∥·∥ v ˆ x ∥ ) , where v x = SimCSE ( x ) and v ˆ x = SimCSE (ˆ x ) .
Perplexity of Generated Text.
Lastly, we evaluate the perplexity of the generated text ˆ x given the prefix x , which is defined as: gen-ppl = 2 f
( D ,θ ) and f ( D , θ ) =
[Equation]
Importantly, the optimal approach should produce text which has a perplexity close to that of the human-written text [14].
A high gen-ppl means the generated text is very unlikely given the prefix, therefore being low quality.
In contrastive, a low gen-ppl means the generated text has a low diversity and gets stuck in repetitive loops [14].
We use the model θ trained with L SimCTG to measure the gen-ppl of different approaches, therefore making sure the numbers are comparable with each other.
4.2 Results
The experimental results on Wikitext-103 are shown in Table 1.
Language Modelling Quality.
From the results, we observe that SimCTG achieves the best perplexity and next token accuracy.
The reason is that, with more discriminative representations, SimCTG is less confusing when making next token predictions, leading to the improved model performance.
On the rep and wrep metrics, the unlikelihood model yields the best result but at the expense of unfavorable performance drops in the perplexity and next token accuracy.
Generation Quality.
Firstly, on the rep-n and diversity metrics, SimCTG + contrastive search obtains the best result, suggesting it best addresses the degeneration problem.
Secondly, the MAUVE score demonstrates that SimCTG + contrastive search generates texts that are closest to human-written texts in terms of token distribution.
Thirdly, among all methods, SimCTG + contrastive search is the only approach that achieves over 0.6 coherence score, showing it produces semantically consistent text with respect to the prefix.
Lastly, the gen-ppl metric also validates the superiority of SimCTG + contrastive search as it obtains notably better generation perplexity comparing with other approaches.
Moreover, from the results of MLE and Unlikelihood baselines, we see that contrastive search still brings performance boost as compared with greedy and beam search.
However, the performance gain still lags behind SimCTG, which demonstrates the necessity of contrastive training.
The underlying reason is that, without using the contrastive objective L CL (Eq.
(2)), the token representations obtained by MLE or Unlikelihood are less discriminative (§6.1).
Therefore, the degeneration penalty (Eq.
(5)) of different candidates are less distinguishable and the selection of output is dominated by the model confidence, making contrastive search less effective.
[Table]
4.3 Human Evaluation
We also conduct a human evaluation with the help of graders proficient in English from a third-party grading platform.
We randomly select 200 prefixes with length of 32 from the test set of Wikitext-103.
For each prefix, we use different models (MLE, Unlikelihood, and SimCTG) with two decoding methods (nucleus sampling and contrastive search) to generate text continuations with length of 128.
To examine the generality of our approach across different model sizes, we include a large size
SimCTG (i.e., SimCTG-large ) which is obtained by fine-tuning the GPT-2-large model that consists of 36 Transformer layers with 20 attention heads.
All generated results, plus the reference text, are randomly shuffled and evaluated by five graders, which results in 9,000 annotated samples in total.
The evaluation follows a 5-point Likert scale (1, 2, 3, 4, or 5) for each of the following features:[-
-]
• Coherence : Whether the generated text is semantically consistent with the prefix. • Fluency : Whether the generated text is fluent and easy to understand.
• Informativeness : Whether the generated text is diverse and contains interesting content.
Table 2 presents the human evaluation results, with the first row showing strong inter-annotator agreements as measured by Fleiss ′ kappa coefficient [12].
Firstly, we see that, directly applying contrastive search with MLE or Unlikelihood model does not yield satisfactory results.
This is due to the anisotropic nature of their representation space as discussed in Section §4.2.
Secondly, the coherence score of Unlikelihood model is notably lower than MLE and SimCTG, suggesting it generates the most unlikely results which is also shown by its generation perplexity (gen-ppl) in
Table 1.
Furthermore, the results of SimCTG + contrastive search significantly outperforms nucleus sampling with different models in terms of coherence and fluency (Sign Test with p-value <
Lastly, SimCTG-large + contrastive search achieves the best performance across the board and even performs comparably with human-written text on the fluency metric (Sign Test with p-value > 0.4).
This reveals the clear generalization ability of our approach to large size models and future work could focus on extending it to models that contain over billions of parameters such as GPT-3 [4].
Open-domain Dialogue Generation
To test the generality of our approach across different tasks and languages, we then evaluate our method on the task of open-domain dialogue generation.
In this task, given a multi-turn dialogue context (where each turn is an user utterance), the model is asked to generate an adequate response that is semantically consistent with the context.
Here, the dialogue context is deemed as the prefix.
Benchmark and Baselines.
We conduct experiments on two benchmark datasets from two languages (i.e., Chinese and English).
For the Chinese benchmark, we use the LCCC dataset [53].
For the
English Benchmark, we use the DailyDialog dataset [24].
We compare the GPT-2 models fine-tuned with SimCTG and MLE.
Specifically, for the Chinese benchmark (i.e., LCCC), we use a publicly available Chinese GPT-2 [61].
9 Same as in Section §4, during training, we use a batch size of 128 and truncate the training samples to a maximum length of 256.
On the LCCC dataset, we train (i.e., fine-tune) the models for 40k steps.
As for the DailyDialog dataset, due to its smaller dataset size, we train the models for 5k steps.
For optimization, we use
Adam optimizer and a learning rate of 2e-5.
For each model, we use four decoding methods, including (1) greedy search; (2) beam search (beam size of 10 ); (3) nucleus sampling ( p = 0 . 95 ); and (4) contrastive search ( k = 5 , α = 0 . 6 ).
Evaluation.
We rely on human evaluation to assess the model performance.
Same as in Section §4.3, we randomly select 200 dialogue contexts from the test set and ask five annotators to evaluate the generated responses plus the reference response in three dimensions: (i) coherence, (ii) fluency; and (iii) informativeness.
The scores follow a 5-point Likert scale (1, 2, 3, 4, or 5)
[Table]
Table 3 shows the evaluation results where the first row shows strong inter-annotator agreements as measured by Fleiss
′ kappa coefficient.
On both datasets, we see that SimCTG + contrastive search significantly outperforms other methods on various metrics, suggesting that our approach is generalizable to different languages and tasks.
It is worth emphasizing that, on the LCCC benchmark,
SimCTG + contrastive search surprisingly outperforms the human performance on the fluency metric, while performing comparably on the coherence and informativeness metrics (Sign Test with p-value > 0.4).
Moreover, even without contrastive training, the MLE model performs significantly better when using contrastive search.
This is due to the intrinsic property of Chinese language model for which making contrastive search directly applicable.
This finding is particularly attractive as it reveals the potential applicability of contrastive search on off-the-shelf (i.e., without contrastive training) language models for certain languages such as Chinese.
Further Analysis 6.1 Token Representation Self-similarity
To analyze the token representations learned by SimCTG, we follow Ethayarajh [10] and define the averaged self-similarity of token representations within a text sequence x as
[Equation]
where h x i and h x j are the token representations of x i and x j produced by the model.
Intuitively, a lower self-similarity ( x ) indicates the representations of distinct tokens within the sequence x are less similar to each other, therefore being more discriminative.
We use texts from Wikitext-103 test set and compute the self-similarity of token representations over different layers for different models.
Figure 2 plots the results averaged over all samples.
We see that, in the intermediate layers, the self-similarity of different models are relatively the same.
In contrast, at the output layer (layer 12), SimCTG’s self-similarity becomes notably lower than other baselines.
We note that the Unlikelihood model also yields more discriminative representations than MLE, but its language model accuracy is lower than MLE and SimCTG as shown in Table 1.
On the other hand,
SimCTG obtains the most discriminative and isotropic representations while maintaining the best language model accuracy, which further validates the clear advantage of our proposed approach.
6.2 The Effect of Contrastive Loss Margin
Next, we analyze the effect of contrastive loss margin ρ (Eq.
To this end, we fine-tune the GPTby varying ρ from 0 .
1 to 1 .
and measure the model perplexity on the Wikitext-103 test set.
Figure 3 plots the results of different ρ along with the result of the MLE baseline.
Note that, when ρ = 0 ,
SimCTG is equivalent to MLE (Section §3.1).
From Figure 3, we see that the contrastive training always helps to improve the perplexity as compared with MLE.
However, when ρ is either too small (e.g., 0 . 1 ) or large (e.g., 1 . 0 ), the learned representation space of the model would be either less or too isotropic, leading to a sub-optimal perplexity.
In our experiments, the most suitable margin ρ = 0 . 5 .
6.3 Contrastive Search versus Nucleus Sampling
Then, we provide an in-depth comparsion between our proposed contrastive search and the current state of the art, nucleus sampling.
To this end, we compare the results of SimCTG using these two decoding methods.
Specifically, we vary the probability p for nucleus sampling and the α (Eq.
(5)) for contrastive search to generate results using prefixes from Wikitext-103 test set.
11 We evaluate the results from two aspects: (1) generation diversity and (2) perplexity of the generated text (gen-ppl).
Both metrics are described in Section §4.1.2.
Figure 4 plots the results of different methods along with the human performance.
For nucleus sampling, when p is small (i.e., p ≤ 0 . 7 ), its generation perplexity is comparable to that of human.
However, the diversity is notably lower than human performance, meaning it stuck in undesirable repetition loops [14].
On the other hand, when p is large (i.e., p ≥ 0 . 95 ), the generation diversity is close to that of human but the generation perplexity is significantly higher.
Such high perplexity means the generated text is very unlikely , therefore being low quality.
As for contrastive search, when α ∈ [0 . 5 , 0 . 8] , it yields generation diversity and perplexity that are both comparable to human performance.
These results demonstrate the superiority of contrastive search as it better balances the trade-off between the generation diversity and perplexity.
6.4 Decoding Latency Comparison
We compare the decoding latency of different decoding methods using SimCTG.
For beam search and contrastive search, we vary the beam width b and the k in Eq.
The latency is measured by generating fixed length text continuations on Wikitext-103 test cases with a batch size of 1 .
In Figure 5, we show the averaged relative decoding latency of different methods.
We see that greedy search is the fastest method and the latency of different methods are generally comparable with each other.
Comparing contrastive search with beam search, when b and k are small (i.e., ≤ 6 ), their latency are nearly identical.
When b and k gets larger (i.e., > 6 ), contrastive search becomes faster.
In summary, these comparison results further verify the practical usage of contrastive search. [Figure] [Table] puzzles.
Buchanan also criticized the controls, calling div. coh.
gen-ppl beam them " too easy " and " too difficult ".
He also criticized the game’s graphics, stating that the game’s graphics were " too simplistic " and " too simplistic ". He also criticized the game’s graphics, stating that the game’s graphics were " too simplistic " and " too simplistic ". He also criticized the graphics, stating that the game’s graphics were " too simplistic " and " too simplistic ". He also criticized the graphics, stating that the game’s graphics were " too simplistic " and " too simplistic ".
He also criticized the graphics, stating that the game’s graphics were" ...
nucleus them " offensively rudimentary ".
After the game was released, Buchanan wrote a number of game reviews about his review of the game, primarily summarizing what he thought the reviewers had enjoyed about it.
Buchanan was pleased with how consistently these reviews awarded the game.
For one, he praised the puzzles, compared them to Nos.
{+1 and 
+}
and enjoyed the construction of the artificial intelligence in the game’s forms.
He found no fault with the puzzles, even in those situations where they seemed repetitive.
According to an Xbox Magazine review, " every time ... contrastive them " unimpressive " and a " nightmare " of an experience to play with players unfamiliar with Tetris.
On the other hand, his opinion was shared by other reviewers, and some were critical of the game’s technical design for the Wii version of Tetris.
In addition, Tintin’s review included a quote from Roger Ebert, who said that Tetris was better than the original game due to its simplicity and ease of play.
Ebert’s comments were included in the game’s DVD commentary, released on March 22, 2010.
It is unclear if any of the video commentary was taken from ... 6.5 Case Study In Table 4, we present generated examples of SimCTG with different decoding methods given a specific prefix.
12 From the results, we see that beam search produces undesirable sequence-level repetitions, resulting in low diversity and low generation perplexity.
On the other hand, in the prefix, the person “Buchanan” criticizes the game.
However, the result from nucleus sampling displays a contradicted semantic, resulting in a low coherence score as well as a high generation perplexity. As for contrastive search, it generates a text that is semantically consistent to the prefix with a proper generation perplexity while obtaining the same diversity as that of the nucleus sampling.
Additionally, it is worth emphasizing that, while the degeneration penalty in Eq.
(5) encourages the model to generate diverse outputs, contrastive search is still able to generate reasonable repetitions as highlighted in Table 4. This is due to the incorporation of model confidence in Eq. (5) which enables the model to repeat the important content (e.g., person names or entity names) from the previous context like humans do. The token similarity matrix of the prefix and the generated text are highlighted in red and yellow.
6.6 Comparison of Token Similarity Matrix
To better understand how contrastive search works, in Figure 6, we show the generated token similarity matrix of SimCTG using beam search and contrastive search. For a better comparsion, we also include the result of MLE using beam search. All results are produced with the same prefix as in
Table 4.
The red and yellow boxes highlight the similarity matrix of the prefix and the generated text.
Firstly, we see that, the MLE + beam search yields a very dense similarity matrix, meaning that its token representations are indiscriminative.
In addition, the high similarity scores in its off-diagonal entries clearly show the degeneration repetitions.
Secondly, for SimCTG + beam search, we observe a desirable similarity matrix of the prefix which is sparse and isotropic.
However, degeneration repetitions still exist in the generated result as shown in Figure 6(b).
Lastly, for SimCTG + contrastive search, the entire similarity matrix is sparse and isotropic, showing that it successfully solves the model degeneration.
These observations are in line with our motivations as described in Section §1.  Conclusion In this work, we show that the degeneration of neural language models stems from the anisotropic nature of their token representations.
We present a new approach, SimCTG , for training the language model such that it obtains an isotropic and discriminative representation space.
In addition, we introduce a novel decoding method, contrastive search , which works coherently with the proposed
SimCTG.
Extensive experiments and analyses are conducted on three benchmarks from two languages.
Both automatic and human evaluations demonstrate that our approach substantially reduces model degeneration and significantly outperforms current state-of-the-art text generation approaches.
Acknowledgments
The first author would like to thank Jialu Xu and Huayang Li for their insightful discussions and supports. Many thanks to our anonymous reviewers, area chairs, and senior area chairs for their suggestions and comments.
A Future Work
For future work, we would like to suggest three research directions based on our study.
• Our proposed contrastive loss L CL in Eq.
(2) is designed to treat all other tokens within the same sequence as negative samples.
However, we do acknowledge that there might be a suitably small fraction of tokens (within the same sequence) that share similar semantic meanings even with different surface forms. We believe the current formulation of the contrastive loss might be further improved by taking this aspect into consideration and we leave it to our future work.
• One limitation of the proposed contrastive search is that it is a deterministic decoding method.
It would be interesting and useful to incorporate a certain level of stochasticity into the decoding process. One plausible approach is to combine contrastive search with stochastic sampling methods. For instance, given the prefix, we could first generate a few tokens (e.g., 1 ∼ 3 tokens) with nucleus sampling. Then, we switch to contrastive search for the remaining steps.
In Appendix L, we provide some preliminary experiment results on incorporating stochasticity into contrastive search. • Our approach is architecture agnostic and can be applied to any generation model. Future research could focus on adapting it to other tasks than open-ended text generation (i.e., constrained text generation), such as machine translation and document summarization.
Neural Text Generation is a core component in many NLP applications. It can be generally categorized into two classes (1) constrained generation; and (2) open-ended generation.
Constrained generation tasks are always defined over a set of (input, output) pairs, where the output is a transformation of the input following specific constrains. Some typical examples include machine translation [49, 2, 29], text summarization [37, 41], and data-to-text generation [55, 47, 45, 57]. As the output is tightly scoped by the input, the generation of repetition and unnaturalness are not that problematic, therefore maximization-based decoding methods such as beam search generally perform well.
Still, different variants of beam search have been explored to further improve the model performance in constrained generation tasks [18, 19, 28, 27]. Open-ended text generation
, on the
other hand, imposes less constrain on the generated text. It aims at producing text that is natural, coherent and informative with respect to the human-written prefix (i.e., context). Several typical applications include story generation [11, 43], contextual text completion [36], and dialogue systems [48, 46].
However, due to the challenges posed by the increased level of freedom, conventional maximization-based decoding methods (e.g., greedy and beam search) often produce undesirable repetition and unnaturalness in the generated text. To alleviate model degeneration, different sampling approaches [11, 14, 30] have been proposed to generate text by drawing samples from less likely vocabularies. Welleck et al. [54] tackled model degeneration from another perspective by introducing unlikelihood objective into the training of the language model.
Contrastive Learning. Generally, contrastive learning methods aim to teach the model to distinguish observed data points from fictitious negative samples. They have been widely applied to various research areas. In the field of computer vision, contrastive learning has been shown to benefit tasks like image [51] and video [39] representation learning.
Chen et al.
[5] proposed a simple framework,
SimCLR, for learning contrastive visual representations.
Recently, Radford et al.
[35] and Jia et al.
[15] applied contrastive learning for the pre-training of language-image models.
In the field of NLP, contrastive learning has recently gained much more attention.
Numerous contrastive approaches have been proposed to learn better token-level [44], sentence-level [31, 25, 13], and discourse-level [42, 21, 1, 20] representations.
Beyond representation learning, contrastive learning has also been applied to other NLP applications, such as name entity recognition (NER) [7], document summarization [26], and knowledge probing for pre-trained language models [32].
Our work, to the best of our knowledge, is the first effort on applying contrastive learning to address neural text degeneration.
We hope our findings could facilitate future research in this area.
C Software Package
In this section, we illustrate the use of the accompanying Python package, available on Github 13 and installable via pip 14 as pip install simctg --upgrade .
Below, we show how to replicate our result in Table 4 with our provided package.
More details can be found in our open-sourced repository 15 .
1 import torch
2 # load the language model 3 from simctg.simctggpt import SimCTGGPT 4 model_name = r’cambridgeltl/simctg_wikitext103’ 5 model = SimCTGGPT(model_name) 6 model.
eval () 7 tokenizer = model.tokenizer 89 # prepare input 10 prefix_text =
# The prefix text in Table11 print (’Prefix is: {}’. format (prefix_text)) 12 tokens = tokenizer.tokenize(prefix_text) 13 input_ids = tokenizer.convert_tokens_to_ids(tokens) 14 input_ids = torch. LongTensor(input_ids).view(1,-1)
1516 # generate result with contrastive search 17 beam_width, alpha, decoding_len = 8, 0.6,18 output =
model.fast_contrastive_search(input_ids=input_ids, beam_width=beam_width, alpha=alpha, decoding_len=decoding_len) 21 print ("Output:\n" + 100 * ’-’)
22 print (tokenizer.decode(output))
Listing 1: Example usage of the SimCTG package
[Table] D Experiments on Different Language Models
In this section
[Table] we further test the generalization ability of our approach with different language models on the Wikitext-103 benchmark.
In addition to the GPT-2-small model (i
[Table]
e. 12 Transformer layers with 12 attention heads) that we consider in Section §4, we include (i) a vanilla Transformers (i
[Table]
e. without any pre-training) with the same parameter size as GPT-2-small; and (ii) a larger pretrained model, GPT-2-large, that consists of 36 Transformer layers with 20 attention heads. The training of different language models follows the same procedure as described in Section §4.
To measure the isotropy of the language model, we include the conicity metric [40] as well as the self-similarity metric (Eq.
A lower conicity or self-similarity indicates the representation space of the language model better follows an isotropic distribution. Table 5 presents the experimental results. We observe that our approach (i.e. SimCTG + contrastive search) performs the best on all evaluated models, suggesting the clear generalization ability of our approach. Another interesting finding is that, for vanilla Transformers and GPT-2-large, the model trained with MLE naturally displays a high level of isotropy.
A similar phenomenon is also observed in language models from other languages, such as Chinese (see Appendix H). In such cases, our proposed contrastive search can be directly applied and yields superior performances.
This further points out the huge potential of contrastive search in other much larger and stronger language models such as GPT-3 [4] and OPT [59]. We leave the rigorous investigation on the isotropic properties of different language models to our future work.
Here, we present a detailed ablation study on the hyperparameters (i.e., k and α in Eq.
(5)) of contrastive search. Specifically, we simultaneously vary the value of k and α . k is chosen from { 5 , 8 , 10 } and α is chosen from { 0 . 4 , 0 . 5 , 0 . 6 , 0 . 7 , 0 . 8 , 0 . 9 , 1 . 0 } .
For evaluation, we report the generation diversity and generation perplexity on the test set of Wikitext-103.
The results are plotted in Figure 7.
We see that, when k is constant, the increase of α generally increases the generation diversity and generation perplexity. When α is constant, a larger k also leads to the increased generation diversity as well as generation perplexity. Nonetheless, for different k , the overall trends are relatively the same and the value of α has more impact on the generated results. In practice, our recommended selection range of k and α are k ∈ [5 , 10] and α ∈ [0 . 5 , 0 . 8] , as these settings produce results that are more similar to human-written texts as judged by generation diversity and generation perplexity.
on the hyperparameters of contrastive search.
[Table] [Table]
In Table 6 and 7, we show the gen-ppl (detailed in §4.1.2) results of different methods as measured by the model trained with MLE and Unlikelihood, respectively. As we use different models to measure gen-ppl, the results in Table 6 and 7 are slightly different from the ones in Table 1.
Nontheless, we can draw the same conclusion as in Section §4.2 that SimCTG + contrastive search is the best performing method as it obtains the generation perplexity that is closest to the human-written text.
Given the human-written prefix, please evaluate the system’s result with respect to the following features: (1) Coherence; (2) Fluency; and (3) Informativeness. In the following, we provide some guidelines regarding how to judge the quality of the system’s result in terms of different features.
G.1 Coherence
This metric measures whether the system’s result is semantically and factually consistent with the human-written prefix. The definitions of different scores are: • [5] : The system’s result is perfectly in line with the semantic meaning defined by the prefix.
And all its content is factually supported by or can be logically inferred from the prefix. • [4] : The system’s result is very related to the prefix but with some minor errors that does not affect its overall relevance with respect to the prefix. • [3] : The system’s result is, to some extent, relevant to the prefix with some errors that display minor semantic inconsistency or contradiction. • [2] : At the first glance, the system’s result seems to be related to the prefix.
But with careful inspection, the semantic inconsistency can be easily spotted.
• [1] : The system’s result is obviously off-the-topic or it is semantically contradicted to the content contained in the prefix. G.2 Fluency This metric measures the fluency of the system’s result. The definitions of different scores are:
• [5] : The system’s result is human-like, grammatically correct, and very easy to understand.
• [4] : Choose this score when you are hesitant between the score 3 and score 5.
• [3] : The system’s result contains minor errors but they do not affect your understanding.
• [2] : Choose this score when you are hesitant between the score 1 and score 3.
• [1] : The system’s result does not make sense and it is unreadable.
This metric measures the diversity, informativeness, and interestingness of the system’s result. The definitions of different scores are: • [5] : The system’s result is very informative and contains novel content.
In addition, it displays a high level of diversity and it is enjoyable to read.
• [4] : Choose this score when you are hesitant between the score 3 and score 5.
• [3] : The system’s result contains some new information and it displays a certain level of diversity.
• [2] : Choose this score when you are hesitant between the score 1 and score 3.
• [1] : The system’s result is dull, repetitive, and does not have new information.
All its content has already been provided in the prefix. Participant Compensation. In each experiment (i.e., open-ended text generation and open-domain dialogue generation), we hire 5 annotators to conduct the human evaluation. For every task, each annotator is paid by $400.
H Self-similarity of Chinese Language Models
We follow the same procedure as described in Section §6.1 to measure the token self-similarity of
Chinese language models.
Specifically, we use the test set of LCCC benchmark and compute the model’s self-similarity. Figure 8 plots the layer-wise token self-similarity of the MLE and SimCTG models. We see that in all layers (including the final layer), the MLE model displays a similar self-similarity with respect to SimCTG. This observation is quite different from what we see from
English language models as shown in Figure 2, where the self-similarities of SimCTG and MLE are notably different in the final layer.
We conjecture that this discrepancy might come from the intrinsic property of different languages.
For English, current state-of-the-art methods always represent the text into subword units, such as BPE [38], and the same subword could be over-shared by many different contexts.
Thus, the representations of distinct subwords become less distinguishable which naturally leads to the anisotropy in their representations.
On the other hand, languages like Chinese are naturally represented by basic units, i.e., characters.
Such natural unit boundary of text alleviates the over-sharing of characters in different contexts.
As a result, even the vanilla MLE objective can obtain a representation space that displays a high level of isotropy.
This isotropic property of Chinese language model is particularly attractive as contrastive search can be directly applied even without contrastive training as shown in Table 3.
In addition, we expect contrastive search could be used on off-the-shelf language models that are trained with MLE in other languages whose texts are naturally tokenized by characters (e.g., Korean and Japanese).
This remains to be rigorously tested in our future work.
[Table]
I Training Efficiency Comparison
In this part, we compare the training efficiency of different methods (i.e., MLE, Unlikelihood, and
SimCTG).
To this end, we compute the total floating point operations (FLOPs) required for the training of different models on Wikitext-103.
The details of training setup are provided in Section §4.
Table 8 shows the results, from which we see that SimCTG is more efficient than the unlikelihood method.
Comparing with MLE, SimCTG only introduces an negligible 1.48% extra computational overhead, which further verifies the practical usage of SimCTG.
J Generated Examples on Open-domain Dialogue Generation
In Table 9, we show some generated responses of our approach (i.e., SimCTG + contrastive search) plus the reference response on examples from the test set of the Chinese LCCC benchmark.
We see that, given the dialogue context, our approach is able to generate responses that are both grammatically fluent and semantically consistent with the dialogue context.
These results further demonstrate the generality of our approach across different languages and tasks.
K More Generated Examples of SimCTG + Contrastive Search
In Table 10, we provide more generated examples of SimCTG + contrastive search based on prefixes from Wikitext-103.
The details of the decoding procedure are described in Section §4.
[Table]
a broad area of disturbed weather
[Table]
became extratropical, which would have caused the storm to turn to the west-northwest in the Gulf of Mexico on April 10.
However, due to the proximity of the cyclone to an area of low pressure
[Table]
Kujira’s winds were less than 10 mph ( 15 km / h ) at the time of landfall.
By April 11, Hurricane Hunters estimated that there were 5,000 people in the area, of whom 1,000 were located in Puerto Rico.
The storm weakened to a tropical storm early on April
[Table]
and dissipated the next day.
Later that
[Table]
The island is composed of a variety of volcanic and sedimentary rock formations from the Devonian period.
At that time the Scottish landmass formed part of the Old result
North Sea, and it was inhabited by an extensive and diverse population of sea-dwelling birds that had lived in the islands for many centuries.
There is evidence of human habitation in the Isles of Scilly and St Albans, and some archaeological evidence is in the form of stone mounds dating to the 12th century.
The most northerly of these is the Murchison Formation, which is located south of the island’s mouth and extends to the northwest as far as Aberdeenshire.
Other sites include the Isle of Man, Isle of Wight, Isle of Orkney, The Loch Ness Monster ...
[Table]
A report published in the American magazine Time a fortnight later described the incident as " a genuine horror story, calculated to make the most alarming of Rhodesian doomsday
[Table]
scenarios ".
In an interview with BBC Radio 4’s Tom Pertwee, he said : " There’s no evidence that anyone in Rhodesia, other than the police, had done anything to provoke it ".
On 20 April 2013, the government announced that it had found no evidence of involvement by Rhodesia in the attack, and would continue its investigation as soon as possible.
The Independent Commission Against Corruption ( ICAO ) was appointed to investigate the incident in December 2013, and is the first body to do so in the UK.
ICAO Director-General Mark Rowley said it was " the most ...
[Table]
The American poet Ezra Pound was introduced to the group in April 1909 and found that their ideas were close to his own.
In particular, Pound’s studies of
[Table]
German Expressionism ( 1891 – 1903 ) helped him develop a critical attitude towards the subject, and he had the opportunity to speak at the first meeting of the group on 25 April 1909, at New York’s
Waldorf-Astoria Hotel.
This was the first time Pound had been invited to speak at a German Expressionist conference, and it proved to be a success.
Pound became an admirer of Expressionism from the outside, and was able to attract many of its members by providing them with information and writing, such as letters to German Expressionists in Germany, as well as photographs of their works ...
[Table]
The Olmec civilization developed in the lowlands of southeastern Mexico between 1500 and 400 BC.
The Olmec heartland lies on the Gulf Coast of Mexico
[Table]
, about 1,500 miles ( 2,500 km ) west of Acapulco County, where there is a population of 1,500 people.
This is the largest city in the state, and it is home to the city’s most famous tourist attraction, the El Centro de las
Amazons, which has been named by Mexican historian José Martínez Sánchez for its beauty and architecture.
The city, along with its surroundings, were important in the development of the Maya civilization, and in later
Mesoamerica and Latin America as well.
Today, Mexico’s ...
[Table]
The
[Table]
at Tres Zapotes in the
[Table]
spurred the first archaeological investigations of Olmec culture by Matthew
[Table]
in
[Table]
[Table]
He showed that the remains of Olmec were in good condition, and suggested that there was something to be said for the fact that they had lived long enough to be recognized as an important part of the culture and its history.
Stirling’s work was the first to establish what was to become known as the Peruvian-Spanish language, or Peru : the language of the people of Peru and Central America.
In doing so, he drew on archaeological evidence and other sources to support his theory of the cultural origins of the Peruvian people.
This is reflected in the name " Peruvian ", which stands for Peruvian ...
L Diverse Contrastive Search
In this part
[Table]
we present a stochastic version of contrastive search (i
[Table]
e., diverse contrastive search) which is described in Appendix A. Specifically, given the prefix with length of 32, we first generate 2 tokens using nucleus sampling with p = 0 .
95 , then we use contrastive search to generate the remaining 126 tokens (i
[Table]
e., 128 generated tokens in total)
[Table]
Table 11 shows three generated results with diverse contrastive search using the same prefix as in
Table 4.
We see that only sampling 2 tokens at the start is enough to produce a diverse set of results.
In future work, we will investigate other more sophisticated extensions of contrastive search.