C O NT: Contrastive Neural Text Generation
Abstract
Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem.
It introduces a sequencelevel training signal which is crucial to generation tasks that always rely on autoregressive decoding.
However, previous methods using contrastive learning in neural text generation usually lead to inferior performance.
In this paper, we analyse the underlying reasons and propose a new Co ntrastive N eural T ext generation framework, C O NT.
C O NT addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects – the construction of contrastive examples, the choice of the contrastive loss, and the strategy in decoding.
We validate C O NT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, datato-text generation and commonsense generation.
Experimental results show that
C O NT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin.
Especially, C O NT surpasses previous the most competitive contrastive learning method for text generation, by 1.50 BLEU on machine translation and 1.77 ROUGE-1 on summarization, respectively.
It achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.
Introduction
Contrastive learning has achieved great success in representation learning [6, 44, 45].
It also attracts enormous interests in neural text generation recently.
By creating positive and negative examples in response to unseen (or erroneous) inputs [23], contrastive learning offers a new solution to alleviate the exposure bias problem [3, 35] – an autoregressive model trained only using the ground truths exhibits inferior generalization performance.
Apart from that, contrastive learning also introduces a sequence-level loss in addition to the conventional token-level language model loss with maximum likelihood estimation (MLE).
This is crucial to most conditional text generation tasks (e.g., machine translation and summarization) which are evaluated on sequence-level metrics (e.g., BLEU [32]).
However, it is non-trivial to get contrastive learning working on neural text generation.
If we simply use from-batch positive-negative samples following simCLR [6], and adopt the InfoNCE loss [13, 45] which ignores the difference between negative samples (§2.2; Naive CL), the improvement over non-contrastive baselines on generation tasks is rather marginal.
Previous work attempts to build better contrastive samples by disturbing the ground truth [10, 23, 30] in the discrete space or the continuous embedding space, but when it comes to text generation tasks, their performance gains are still far from satisfactory.
<bos> leave that up to us <bos> um , my brand ?
<
bos> this is frightening <bos> they sell certificates
f r o mba t
c h s a m p l e s überlassen sie das ruhig uns
Encoder input
Encoder-Decoder
Ground truth
Decoder input of naive method
Decoder input of our method <bos> leave that up to us <bos> leave that quiet to us <bos> leave that quiet down <bos> let's take that quietly
Ground truth s e l fgene r a t ed s a m p l e s from-batch samples
In this work, we propose a new contrastive neural text generation framework, C O NT.
C O NT does three different things from previous frameworks that make suboptimal use of contrastive learning.
First, C O NT samples contrastive examples from its own predictions (e.g., through beam search algorithm).
This training procedure exposes the model to its mistakes in the inference stage and effectively alleviate the exposure bias problem.
We show a comparison between negative samples in C O NT and in Naive CL in Figure 1.
Second, we use a N-pairs contrastive loss which gives a fine-grained treatment to the contrastive examples based on their sequence-level scores (e.g., BLEU).
It allows the model to fully leverage the supervision from the ground truth example (and its own generated examples) to learn a better sequence-level distance function between the source and the target representation.
Third, we directly incorporate the learned sequence similarity score from the distance function into the inference stage.
This helps the model to find a better global configuration, than merely follows the language model likelihood objective in decoding.
We validate C O NT on various important conditional language generation tasks (§4.2), including machine translation, summarization, code comment generation, data-to-text generation, and commonsense generation.
Extensive experiments demonstrate that
C O NT greatly improve the conventional
MLE baselines and significantly outperforms all previous contrastive generation models.
C O NT establishes new state-of-the-art results on summarization, code comment generation (without external data), and data-to-text generation.
Particularly, on data-to-text generation and commonsense generation, C O NT achieves on-par performance with the powerful large pre-trained models: T5-large,
T5-3B [36] with only the base version of T5 while maintaining the efficiency of lightweight models.
Background 2.1 Neural Conditional Text Generation
A neural sequence-to-sequence model [43] M =
( f, g ) generates the target sequence conditioning on a source sequence, where f and g denote the encoder and decoder, respectively.
It is typically trained using the language model objective with the maximum likelihood estimation (MLE).
Given a source sequence x = { x i } Mi =0 and its target sequence y =
{ y i } Ni =0 , we minimize the following negative log likelihood (NLL) loss:
[Equation]
At training stage, it predict the next word based on previous ground truth input y <t ∈ y , but at inference stage, tokens of y <t are predicted by itself, this introduces the exposure bias .
2.2 Naive Contrastive Learning for Text Generation
Contrastive text generation introduces a contrastive term in addition to the original NLL loss.
In
Naive CL, we simply follows SimCLR [6] and use from-batch negative samples B in the InfoNCE loss [13, 45]: überlassen sie das ruhig uns
Encoder
Decoder <bos> leave that up to us beam search z y z y' z y ''
z B pair-wise contrastive loss
Training obj <
bos> leave that quiet to us, p =0.
<bos> let's take that quietly, p =0.
z
y
z x z y ''
z y'
Decoding obj from-batch samples
z y' z y '' z x sim=0.
sim=0.
score= 0.48 + 0.
score= 0.53 + 0.
The feature representations come from pooling the output of the encoder (source sequence) or decoder (target sequence).
Our training objective is obtained by comparing by all contrastive samples in pair.
The decoding objective not only considers the likelihood of each sequence, but also the sequence similarity score modeled in training.
[Equation]
where z x , z y , z y ′ ∈
R d denote the vector representation of input x , ground truth y and negative sample y ′ ∈ B , respectively.
τ is the temperature and cos( · , · ) defines the cosine similarity.
Intuitively, the contrastive loss L NCE seeks to learn a similarity function that drives the distance between the source sequence representation z x and its ground-truth target sequence representation z y closer.
Method
In this section, we present our new contrastive neural text generation framework, C O NT .
C O NT advances the Naive CL (§2.2) in three aspects.
First, C O NT uses negative examples from its own predictions (§3.1) to construct the set B .
Second, C O NT replaces the InfoNCE loss (Eq.2) with a
N-pairs contrastive loss (Eq.3) which leverages a finer-grained supervision given by the the sequencelevel scores of all pairs (§3.2).
Third, C O NT incorporates the learned similarity function into its inference score directly (§3.3).
An overview of our approach can be found in Figure 2.
3.1 Contrastive Examples from Predictions
Instead of only using contrastive examples from the same batch [6], we propose to add new contrastive examples from the model’s own predictions.
Kalkstein et al.
[18] point that using diverse contrastive samples helps the generalization ability of the model.
Therefore, we use the diverse beam search algorithm [49] to create contrastive examples from the top-K list of the model’s lastest predictions and then append them to the from-batch samples to form the contrastive examples.
A warm-up stage where the model is only supervised by L NLL is recommended as it guarantees the quality of the examples from the model’s prediction.
These self-generated contrastive examples alleviate the model’s exposure bias.
Besides, with the model’s performance improving gradually, this approach creates high-quality hard negative examples that is known to be important in contrastive learning [16, 37].
3.2 N-Pairs Contrastive Loss
One major drawback of the InfoNCE loss is that it has the same treatment for all negative samples.
In text generation, this means that the relative difference between the ground truth and the contrastive examples is ignored, while this can be easily quantified using a sequence level score (e.g. BLEU) and the quality of these contrastive examples varies.
To mitigate this problem, we propose to employ a pair-wise margin loss.
We first rank all the contrastive examples based on an oracle function o ( · , y ) , which computes a sequence-level score with the ground truth y .
Given a input sequence x , the ground truth y , and a set of K contrastive samples B = { y 1 , y 2 , · · · , y K } , we can create a series
I
H x ← ˆ f b ← beam size balance y 1: b , P b y , H b y = B EAM S EARCH (ˆ g b )
4: z x , z 1: b y ←
( H x ) , Avg ( H b y ) function 5:
D y 6:
= b { of example pairs ( y + , y − ) ∈ P , where + and − are determined by their ranks.
The contrastive learning objective is formulated as a margin loss according to their cosine similarity to the source representation z x :
[Equation]
We further set ξ = γ ∗ ( rank ( y − ) − rank ( y + )) following Zhong et al.
[57] to reflect the quality difference in these pairs, where γ is a hyperparameter controlling the the strength.
Full details of the training algorithm can be found in Algorithm 2, Appendix B.
3.3 Inference with Learned Similarity Function
Previous inference algorithm for contrastive text generation method [23] usually remains the same with non-contrastive approaches.
In C O NT, we incorporate the similarity function learned in the
N-pairs contrastive loss into the decoding stage.
Despite such a inference objective can be generalized to other contrastive learning methods as long as the vector representations for source and target sequence pair exist, the design of C O NT can better make use of the learned similarity function (§4.3).
The decoding objective in CoNT is to find the sequence y ∗ that maximizes both the learned similarity score and the conventional language model likelihood:
[Equation]
where z x , z ˆ y ∈ R d is the vector representation of x , ˆ y , and α is the hyperparameter that balances the contribution of each term.
In most cases, α can be directly set to 0.5, tuning α on the validation set will usually get better results.
Algorithm 1 illustrates the inference stage in CoNT in details.
The relationship between different modules of C O NT is summarized in Figure 3.
[Figure]
Experiments
We experiment C O NT on 5 downstream tasks with 10 different benchmarks.
Our contrastive learning framework supports most sequence-to-sequence models at multiple scales.
Concretely, we experiment
C O NT on 4 kinds of base models: (a) Transformer-samll (60M) [47] and transformer-base (220M), (b) T5-small (60M) and T5-base (220M) [36], (c) CodeT5-base (220M) [51] , (d) PEGASUS-large (560M) [55].
Details of our experimental setup for each benchmarks can be found in Appendix C.
On WMT’16 Ro-En (machine translation) and XSum (summarization) which are also used in previous contrastive text generation frameworks [23], results show that C O NT is able to substantially improve the MLE baseline and clearly outperform all previous contrastive baselines by a large margin.
We also build C O NT on state-of-the-art (SOTA) baselines: PEGASUS-large (summarization), CodeT5base (code comment generation) and achieve new SOTA.
Moreover, on data-to-text generation and commonsense generation, C O NT also shows its superior performance over strong MLE baselines.
4.1 Baselines
Naïve CL [6]: Naive CL denotes the naive contrastive learning approach that treats the ground truth as the positive sample and the target sequences from the same mini-batch as the negative examples.
The training object of naive CL takes the form of Eq.
We also implement Naive CL with N-Pairs contrastive loss, it can be viewed as an ablation study when setting beam size of C O NT to 0 during training.
SSMBA CL [30] : Compared with naive CL, SSMBA builds more positive samples via disturbing the ground truth in the discrete space.
Concretely, SSMBA first randomly masks 25% tokens in the target sequence and then reconstructs the ground truth with a masked language model BERT.
Dropout CL [10] : Dropout CL enhances the positive samples by using dropout mechanism on the target sequence.
We use the default dropout rate of standard transformer decoder [47] and input the ground truth to the decoder twice.
CLAPS [23] : CLAPS is previous the best contrastive learning framework for conditional text generation task.
In order to provide more challenging contrastive examples, CLAPS propose to simultaneously create extra positive and negative pairs by adding perturbations to the ground truth sequence in the continuous embedding space.
C O NT (this work):
C O NT is the contrastive neural text generation framework proposed in this work.
We implement its InfoNCE version by treating ground truth as positive sample and self-generated samples are also treated as negative samples.
4.2 Quantitative Results
Machine Translation For machine translation, we evaluate C O NT on WMT 2016 Romanian-toEnglish translation task (WMT’16 Ro-En), WMT 2014 English-to-German translation task (WMT’
En-De) and IWSLT 2014 German-to-English translation task (IWSLT’14 De-En).
We use BLEU as the evaluation metric.
Results in Table 1 (rows with gray background) indicates our model
C O NT significantly improves the traditional maximum likelihood estimation training and inference framework.
On WMT’16 Ro-En, C O NT outperforms previous the best contrastive learning approach
CLAPS by 1.50 BLEU and exceeds the MLE baseline by 2.70 BLEU with the same base model
T5-small.
We also compare the infoNCE loss used in previous methods with the N-Pairs margin loss described in Eq.
Results show that the N-pairs contrasting samples generally works better than dividing all samples into predefined positive-negative categories.
Similar to CLAPS and Naive CL, only incorporating contrastive learning into training improves the performance of T5-small baseline on WMT’16 Ro-En to 30.55 (+ 2.34 ) BLEU.
If we further add learned target-source similarity as decoding target as Eq. 4, the result is further boosted to 30.91 BLEU.
We observe that the benefits of introducing sequence similarity into inference is more obvious on IWSLT’14 De-En – the additional decoding target improves the vanilla beam search algorithm up to 0.86 BLEU.
Summarization For summarization, we use the XSum [28] dataset collected from BBC News whose reference summaries are provided by human writers.
We also evaluate C O NT on a multidocument summarzation dataset multi-news [9] consisting of news articles from the site newser.com.
Compared with the common summarization task, multi-document is more challenging where the model need to automatically summarize several articles and usually has to handle long input sequence
[Table] [Table]
-L
RR-
R-L
T5-small 36.
T5-
[Table]
CL 36.
T5-Dropout CL 36.
[Table]
-CLAPS † 37.
[Table] [Table]
CL 36.
[Table] [Table]
[Table]
Previous SOTA
[Table] [Table]
[Table]
PEGASUS (base) ∗ 39.
PEGASUS (large) ∗ 47.
[Table] [Table]
[Table] [Table]
and target sequence.
Experimental results are in Table 2.
The first block includes the performance of different contrastive frameworks with T5-small.
On XSum, it shows that our proposed model strongly outperform previous contrastive frameworks by about 2.0 ROUGE-1 score.
We also illustrate our method is not restricted to the small model.
By employing C O NT on state-of-the-art base model
PEGASUS, it is able to establish new state-of-the-art on the two summarization benchmarks.
Code Comment Generation Code comment generation aims to generate an English description for a function-level code snippet.
We test our method on two widely used datasets Java and Python from the CodeXGLUE benchmark [27].
Results are shown in Table 3.
Our model is built upon stateof-the-art pre-trained model on program language model CodeT5-base.
CodeT5-Dual-Gen means they further involve a comment-to-code task which is the best model on Python and Java without using external data.
We also include the results of earlier strong pre-trained baselines: PLBART
[Table]
[Table] [Table]
and CodeBERT.
We also report some data augmentation methods in the third block of Table 3.
CodeT5-Multi-Task makes use of training datasets of other program languages.
REDCODER [34] uses retrieval to enhance the task with open-source code base and achieve the bset results on this task.
Our model is orthogonal to these methods and clearly outperforms all baselines without external data.
Data-to-text Generation Data-to-text generation aims to produce text from non-linguistic input.
The first benchmark we use is WikiBio [22] consisting of biography pairs from English Wikipedia where the infobox is treated as input sequence, and the target sequence is the first sentence of the biography.
Totto [33] is also collected from Wikipedia whose input is a table with its highlighted cells and target sequences are professionally annotated by human.
Results on WikiBio is shown in Table 4, the performance of some popular baselines (first block) are token from [25].
R2D2 [2], using XLNet [53]-large as base model, is previous state-of-the-art model on WikiBio.
We experiment
C O NT on T5-small, and results show that we exceed R2D2 by about 0.94 BLEU.
The test set and dev set of Totto are both split into two parts - overlap and non-overlap.
The non-overlap part contains out-of-domain samples from the training set.
The test set of Totto is invisible and we report the results on dev set and test set by the feedback of the Totto authors.
We also add PARENT [8] and
BLEURT [40] as evaluation metrics.
PARENT is a word-overlap based metric that designed to evaluate the factual accuracy of generation results.
BLEURT is trained under human supervision and correlates well with human judgement.
As we can see from Table 5, by comparing our model with different T5 variants, we show that C O NT is able to greatly outperform the large version of T5 even built on a T5-base model.
Compared with previous state-of-the-art model T5-3B, our model still show advantage in PARENT and BLEURT.
Commonsense Generation
The task of commonsense generation aims to explicitly test the ability of machines on commonsense reasoning.
The source sequence consists of a set of concepts and the target sequence is a fluent sentence mentioning all the input concepts.
We evaluate C O NT
[Table]
[Table]
on the CommonGen [24] benchmark with a hidden test set and the final results is obtained with the help of the authors of CommonGen.
In addition to the mostly used metrics, CIDEr [48] and
SPICE [1], concerning to evaluating semantic faithfulness, are highlighted by the leaderboard of commonGen.
In Table 6, we demonstrate that the lightweight T5-base model is able to greatly benefit from our contrastive learning framework.
Moreover, C O NT not only surprisingly outperforms its
MLE baseline but also surpass the large version of T5 in terms of CIDEr and SPICE metrics.
Advanced Evaluation Metrics
Considering the training efficiency of C O NT, we mainly select the lexical matching metrics as oracle measurement function.
To verify that the improvement brought by
C O NT is not due to the over-fitting of lexical matching metrics, we further evaluate generated text with advanced metrics based on neural models:
BERTScore [56] and BLEURT [40].
For BERTScore, we use their roberta-large_L17_no-idf_version and for BLEURT we use the default setting provided on their github 4 .
Results are shown in Table 7.
The base model used on IWSLT’14 De-En is transformer small and on the other datasets we select T5 as the base model.
For all datasets C
O NT also make non-trivial improvements in terms of the two neural metrics.
Particularly, C O NT improve the results of MLE model on IWSLT’14 De-En by 0.03 BLEURT and improve the results of MLE model on XSum by 2.68 BERTScore.
To get more accurate and convincing results, we also conduct a ranking based human evaluation on two mainstream tasks: machine translation (IWSLT’14 De-En) and text summarization (XSum) with 60 samples for each tasks.
Following Cheng and Lapata [7], we hired 2 annotators asking them to rank the given candidate output based on fluency, coherence, and their personal preference ( rank these systems 1st, 2nd, and so on) and we calculate the average ranking.
For each sample, there are four candidates consist of a human-written reference, a sequence from MLE model, a sequence from
Naive CL, and a sequence from C O NT.
Table 8 shows the results of our human evaluation.
Generally
C O NT outperform all baseline systems according to the average ranking.
4.3 Discussion
Discrimination of Hard Negative Samples To deeply look into the learnt representations, we visualize target sequences, that is trained by MLE, naive CL and C O NT on IWSLT’14 De-En, with the t-SNE algorithm [46].
The visualized sequences consist of three groups of target sequence: a) batch targets that is mostly unrelated to ground -truth target; b) beam search hypothesis that could be of high/low quality; c) ground truth target.
As can be seen in Figure 4a, the representations trained by
MLE are distributed almost uniformly in the vector space, and there are no clear boundary between one group to another.
With naive CL, we find there is clear boundary between batch tokens and
[Table]
[Figure]
others.
Naive CL does contribute to discriminating related sequences with unrelated ones, but it still cannot distinguish hypothesis of high quality from the others.
Even without contrastive learning, the generation model trained has already pulled from-batch samples away from the ground truth and the naive contrastive learning procedure is only to make the margin more obvious.
As for C O NT , a set of hypotheses of low quality are excluded from the neighborhoods of the ground-truth targets.
The experimental results verifies that C O NT enables better representations in sequence generation.
Sequence Likelihood and Sequence Similarity We also perform ablation study on our ranking objective and self-generated negative samples on IWSLT’14 De-En with Transformer-small as base model.
For the ranking objective, we compare our N-pairs contrastive objective with InfoNCE with increasing α .
Figure 5a that N-pairs contrastive loss consistently outperform InfoNCE.
With α ∈ [0 . 3 , 0 . 7] , N-pairs contrastive loss can benefit target-source representation similarity, a sequencelevel score in inference process, while InfoNCE cannot.
For the sampling strategy, we compare C O NT with naive CL and vanilla MLE.
C O NT use self-generated hypotheses as negatives while
[Figure]
naive CL only use samples within the same mini-batch.
Figure 5b shows that contrastive learning with self-generated hypotheses is more effective than using batch samples.
With α ∈ [0 . 3 , 0 . 7] , C O NT gains about 1 . {+
+}
BLEU improvements, while naive CL is only improved with less than 0 .
BLEU.
Related Work
Contrastive Learning Contrastive learning [6, 13, 39] aims to learn a better representation via contrasting positive and negative samples.
It also has been widely used in the field of natural language processing [10, 14, 20, 21, 59].
Jiang et al.
[15] show that contrastive learning helps learn a robust pre-trained model and Lee et al.
[23] first introduce contrastive learning into text generation to mitigate the exposure bias problem.
They propose an adversarial method to build more challenging positive-negative samples in additional to the from-batch samples.
SimCTG [42] is also a contrastive framework on text generation.
Our work differs from SimCTG from motivation and method.
They introduce contrastive leanrning mainly to encourage diversity which is important in dialogue systems.
And they perform token-level contrastive learning while our method focus on sequence-level contrastive examples.
Adopting binary supervision in contrastive loss is originally proposed in FaceNet [39] which helps learn the face recognition of the same person in various positions and angles.
Given an anchor face image x , a positive sample x + (usually the same person) and a negative sample x − from other people, the triplet loss makes x + become close to x and maximize the distance between x and x − .
The pair-wise contrastive loss has also been widely used in metric learning Chen and Deng [5], Kim et al.
[19], Wang et al.
Sohn [41] extend the triplet loss to multi-class and multi-pair.
Recent work thinks the margin value between samples should not be fixed.
Zhou et al.
[58] divide the sample set into multiple subsets and assign different margin value to different subsets.
Ha and Blanz [12], Zhong et al.
[57] suggests dynamically adjust the margin value via a determination metric.
Post-generation Re-ranking Models Post-generation re-ranking re
-score the multiple output sequences via training another re-ranking module.
Noisy Channel Modeling (NCM) [29, 54] is a widely-used re-ranking scheme for neural machine translation.
NCM parameterizes the noisy channel probability with a sequence-to-sequence model.
There also various structures to instantiate the re-ranking module: Gulcehre et al.
[11] select the candidate with a language model, Bhattacharyya et al.
[4] leverage an energy-based model in NMT and Liu and Liu [26], Salazar et al.
[38] re-score candidates with masked language models such as BERT.
Despite this paradigm achieves impressive results while having a large size of candidate sequences, most of post-generation re-ranking systems trade efficiency and simplicity for accuracy.
Conclusion
We introduce a new contrastive neural text generation framework called C O NT.
It models an additional contrastive learning objective to provide a sequence-level supervision for auto-regressive neural text generation models.
We explore three shortcomings that limit the development of contrastive learning on text generation tasks.
Results on five generation tasks with ten different benchmarks show that C O NT not only clearly beats all previous contrastive generation models, but also boosts the performance of state-of-the-art large models to a new level.
C O NT practically does not have a negative impact on decoding speed.
Nevertheless, C O NT suffers from the training inefficiency problem.
In general, the total training time of C O NT is about 2 ∼ 4 times more than that of a MLE based model.
A detailed discussion and some speed-accuracy trade-off tricks can be found in Appendix B.
Speeding up the training stage without losing accuracy is the next important step to improve C O NT.
Acknowledgement
We would like to thank the anonymous reviewers for their valuable advice.
This research was supported in part by the joint research scheme of the National Natural Science Foundation of China (NSFC) and the Research Grants Council (RGC) under grant number N HKU714/21.
A Overview
In the supplementary materials of this work, we first discuss the main limitation of C O NT.
After that, we describe the detailed experimental setup for the 10 different benchmarks.
Finally, we randomly present some generation examples of C O NT on machine translation and summarization.
B Limitation
C O NT practically does not have a negative impact on decoding speed.
Compared with the decoding algorithm used in the inference stage of conventional generation models, the additional operations brought by C O NT are reflected in line 4 and line 5, Algorithm 1.
The two operations can be efficiently calculated on GPUs.
However, an obvious disadvantage of C O NT is the sacrifice of training efficiency.
In general, the total training time of C O NT is about 2 ∼ 4 times more than that of a MLE based model.
We show the pseudo code of our training procedure in Algorithm 2.
As can be seen from this algorithm, there are three main factors that harm the training speed of C O NT:
(i) a pre-train stage to ensure meaningful contrastive samples (line 3 in Algorithm 2), (ii) the involvement of tokenby-token decoding in training (line 7 in Algorithm 2) which can hardly benefit from the parallel computing power of GPU, and (iii) the calculation of oracle measurement (the nested loop in line 9 ∼ 11 Algorithm 2).
Regrading the last issue, if we use some lexical matching metrics such as
BLEU or ROUGE that always need calculated on CPU, the training speed will be obviously slowed down.
During waiting for the results from CPU, the GPU utilization will decrease to 0.
We solve the problem with by placing the nested loop on GPU where candidate samples and the ground truth are represented by a long type tensor and the similarity between two sequences are calculated by matrix multiplication which significantly improve the efficiency.
The GPU version of getting oracle measurement will also be released along with our source code.
As for the first issue and the second issue, we haven’t found good alternatives yet so that we advise two ways to make a speed-accuracy trade-off at the following parts.
Algorithm 2 Contrastive Text Generation:
Given a generation dataset <
X , Y > a randomly initialized
-decoder model M
=
( f, g ) ; return contrastive generation model.
: procedure W ARM U P ( M ) 2:
Update the parameters of randomly initialized M with ∇ θ L nll until convergence 1: procedure B EAM S EARCH ( g , H X , b ) ▷ beam search algorithm 2: return Text, likelihood, logits of the b hypotheses 1: procedure T RAIN ( M , < X , Y >) 2: θ ← Parameters of M , b ← beam size 3:
W ARM U P ( M ) 4: while not convergence do 5:
X 1: k , Y 1: k ← A minibatch of k datapoints from < X , Y > 6:
H 1: kX ← f ( X 1: k ) , H 1: k Y ← g ( H 1: kX , Y 1: k )
▷ outputs from the encoder and decoder 7:
Y ′ 1: k, 1: b , P 1: k, 1: b Y ′ , H 1: k, 1: b Y ′ =
[Equation]
EAM S
[Equation]
( g, H 1: kX , b ) 8:
Y ′ 1: k, 1:( b + k ) ← Append b self-generated samples to Y 1: k 9: for i ∈ 1 , 2 , . . .
, k do 10: for j ∈ 1 , 2 , . . .
, , b + k do 11:
Do oracle measurement o
( Y ′ i,j , Y i ) for each element Y
i,j in Y ′ 1: k, 1:( b + k )
L ctr ← Get pair-wise contrastive loss 12: update parameters using ∇ θ
( L nll + L ctr ) 13: return M
Small Beam Size
The first trick is to reduce the proportion of self-generated samples.
With the increase of beam size, the time consumed by beam search will increase significantly.
Remaining the maximum number of contrastive examples for each input unchanged, we can adjust the ratio of selfgenerated samples and from-batch samples.
For IWSLT’14 En-De benchmark with transformer small, the default maximum number of contrastive samples is set to 32 and the proportion of self-generated samples is 75% (settings of other benchmarks are shown in Table 10).
Relationship between the
[Figure] [Figure]
[Figure] [Figure]
P U s ec o nd s
Training time
[Figure] [Figure]
training time of each iteration and the proportion of samples returned by beam search on IWSLT’
De-En benchmark can be seen in Figure 6a 5 .
Totally using self-generated samples will double the training speed of the naive contrastive text generation method and will do not further boost the improvement in performance.
Reducing the rate of self-generated samples to 50% still resulting in 1.0 BLEU superior to the baseline while saving about 1.0 GPU seconds per iteration compared with totally using self-generated samples.
Early Stop Another way to save training time is early stop.
We can see in Figure 6b, on IWSLT’
De-En, the declining trend of the contrastive loss on validation set allowing us to perform early stop in training.
The contrastive loss on valdation set drop rapidly at the first 10k steps, and this decline will slow down in the following steps.
In our experiments we train this model for about 40k steps, and early stop after 10k steps is also enough to improve the MLE baseline by 0.8 BLEU but saving 3/4 training time.
C Experimental Setup
In this section, we will introduce more details about experiments and datasets.
Our experiments on IWSLT’14 De-En and WMT‘14 En-De use fairseq [31] framework.
Experiments on other datasets are run with transformers [52].
Table 9 gives an overview of the number of instances in train/validation/test set and its source.
The test set of Totto [33] and CommonGen [24] is invisible.
We get these results by submitting our generation results to the leaderboards 67 .
[Table] [Table]
C.1 Hyperparameters Brought by Contrastive Learning
Compared with MLE based models, there are three additional hyperparameters introduced by C O NT.
The first one is the maximum number of contrastive samples m for an input sequence, the second one is margin strength γ (defined in Section 3.2) and the third one is the balance factor α .
γ is set to 0.01 for all datasets.
We tune α on the validation set from [0 . 2 , 0 . 3 , 0 . 5 , 0 . 7] .
We set the m toon IWSLT’14 and set m to 16 on other benchmarks considering of efficiency.
Actually, increasing the number of contrastive samples will continuously boost the performance.
On all benchmarks, the beam size for diverse beam search [49] used in training is 0 .
75 ∗ m and the number of groups of diverse beam search is the same with beam size.
Details can be found in Table 10.
Before adding contrastive learning, we pretrain the generation model with only negative log likelihood loss until the validation loss no longer decreases.
All experiments are done on 4 NVIDIA Tesla A100 GPUs.
C.2 Machine Translation
For WMT’16 Ro-En, We use the Adafactor optimizer following [36] to finetune transformer-small and T5-small with learning rate = 1 × 10 − 3 .
The pre-trained checkpoints of T5 are provided by transformers 8 We limit the maximum input/output length to 128.
Validation step is performed every 2000 training steps.
We train our model for 2 epochs and get the best model at step 8000.
It takes about 2 hours on 4 NVIDIA Tesla A100 GPUs with a batch size of 32.
The dimension of hidden state of small-scale model is 512.
So the dimension of representations from the encoder and decoder z is as the same.
At inference stage we set the length penalty to 1.0.
Previous work mainly report their results on IWSLT’14 De-En and WMT’14 En-De based on fairseq [31] library 9 .
We also implement C O NT on the two datasets with fairseq.
For IWSLT’
De-En, we use the small setup of the Transformer model.
The model has 6 layers where model dimension for each layer is 512 and feed-forward dimension is set to 1024.
The batch size is up to
4000 tokens and we update our model every 4 backwards.
On WMT’14 En-De, we use the base setup of the Transformer model where the dimension of feed-forward layer is set to 2048.
The embedding for decoder input and output is shared.
The batch size is also set to 4000 tokens but we update every 20 backwards to simulate large batch size which is very crucial to WMT’14 benchmark.
In addition, we find that open the dropout module in decoder during inference will help the performance for
C O NT on WMT’14 En-De.
We train our model for 20 epochs on IWSLT’14 De-En and 10 epochs on WMT’14 En-De.
We use
GPUs for the model training.
The average running time for IWSLT is about 8 hours and for WMT is around 32 hours.
We use FP16 to accelerate our training.
Other settings are the same with the default settings recommended by the instruction of fairseq official site to re-produce the neural machine translation results 10 .
For both IWSLT’14 De-En and WMT’14 En-De, we use Adam optimizer with learning rate 5 × 10 − 4 with the inverse sqrt learning rate scheduler to optimize the models.
C.3 Summarization
We use the Adafactor optimizer with learning rate = 1 × 10 − 3 to finetune T5-small model and
Pegasus-large [55] model.
For XSum [28], we limit the input length to 512 and output length to 64.
The input length is extended to 1024 and the output length is extended to 300 for multi-document summarization benchmark multi-news.
The length penalty for XSum is set to 0.8 while for multinews, which has longer target sequence, the length penalty is set to 2.0 We use a batch size of 32 for small-scale model and 4 for large-scale model.
We train our model until the validation loss do not decrease.
The total training hours using 4 GPUs is about 6 hours for small model and 12 hours for large model.
C.4 Code Comment Generation
We use the state-of-the-art code comment generation model CodeT5 [51] as our base model.
We download their pre-trained checkpoint from transformers.
we truncate the input length to 512 and output length to 64.
The two benchmark is sensitive to batch size and learning rate.
Therefore, we use a smaller learning rate 1 × 10 − 4 with Adafactor optimizer.
The batch size is set to 8 and other settings are the same with the origin paper of CodeT5 11 .
We train C O NT on the two benchmark for about 4 hours on 4 GPUs.
The length penalty is set to 0.6 during decoding.
C.5 Data-to-text Generation
The input of data-to-text generation tasks is structured data (e.g., table, graph).
To input the structured data into a sequence-to-sequence model, we should first the linear the input.
We linear the input for
WikiBio following Liu et al.
For totto, we use preprocess the dataset following the instruction of the official site 13 .
We use the T5-small model as base model for WikiBio and T5-base for Totto.
We limit the input length to 512 and output length to 128.
The length penalty for WikiBio and Totto is set to 2.0.
We train our model for about 24 hours on 4 GPUs with a batch size of 32.
C.6 Commonsense Generation
For commonsense generation task, we use the popular benchmark CommonGen Lin et al.
The input of CommonGen is a set of concepts and the output is a fluency sentence mentioning all concepts in the source side.
We concatenate these concepts with ’,‘ as separator.
We use the base setup of the T5 model for CommonGen [24].
The maximum input length for source and target is limited to 64.
Other settings are the same with the settings of Totto.
We train our model for 1 epoch upon the checkpoint pre-trained with negative log likelihood loss.
Since the scale of the dataset is samll, it only takes about 0.5 hours training to convergence on 4 GPUs.
D Case Study
We show some randomly selected examples from IWSLT’14 Germanish-to-English translation task and XSum which aims to summarize a news article in table 11,12,13.
[Table] [Table]
gewissermassen überflügelt uns
[Table]
technik also.
[Table]
so in a sense, it’s getting ahead of us.
[Table]
so, in a sense, technology overwhelms us.
[Table]
so, to some extent, technology overrivers us.
[Table]
erzählen sie mir über die "warum" phase–was bringt sie
uns?
[Table]
tell me about the "why" phase – what does that do for us?
[Table]
tell me about the "why" phase – what does it bring us?
[Table]
tell me about why – what does it bring us?
[Table]
und wir können auf diese sehr einfache weise navigieren.
[Table]
and we can just navigate in this very simple way.
[Table]
and we can navigate in this very simple way.
[Table]
and we can navigate this very simple way.
[Table]
und wenn ich jemandem sage, gib mir mal salz oder pfeffer, dann wird er beim rechten erstmal überlegen, wo ist was drin.
[Table]
and if i say to someone, pass me the salt or the pepper, they’d have to first think about what’s in what with the right one.
C O NT: and when i say to someone, i’ll give me salt or pepper, they’ll put it on the right for the first time, where is it in.
MLE:
and when i tell someone, give me salt or pepper, they’ll be thinking about where is there?
Germanish: und problem , das ist nicht nur ein technisches problem, es kann auch ein gesellschaftliches problem sein, es kann auch einfach ein zugangsproblem sein , was wie dinge vereinfachen, also eine beliebige problemstellung , eine frage aufzuwerfen, und wie kann man das anders oder wie kann man das besser machen.
Ground
[Table]
and a problem, so not only a technical problem, it can also be a social problem, it can also just be an access problem that simplifying things, so any way of looking at a problem, of posing a question, asking how you could do something differently or better.
C O NT
[Table]
and the problem is, it‘s not just a technical problem, it can also be a social problem, it can be a problem as well as a problem of accessing how things simplify, which is an arbitrary problem of asking a question, and how do you do it differently, or how do you do it better.
MLE:
and the problem is, it‘s not just a technical problem, it can also be a social problem, it can be a problem as well as a problem of accessing how things simplify, which is an arbitrary problem of asking a question, and how do you do it differently, or better.
[Table]
home side dominated the closing stages but could could not deny Rovers, who remain in fifth place.
Rovers remain level with Airdrieonians, who drew at home with Forfar Athletic.
Ground Truth: Scottish League One leaders Dunfermline Athletic were held at home by Albion Rovers but still moved 11 points clear at the top of the table.
C O NT: Tranmere Rovers slipped to the bottom of Scottish League One as they were held to a draw by play-off chasing Pars.
MLE: Dundee Rovers dominated the Scottish Championships with a 2-0 win over Forfar Athletic.
Document: Reports in France suggest Toulon coach Diego Dominguez’s job is under threat, and Lancaster, 46, is viewed as a potential successor.
He left his role with England after their exit from the 2015 World Cup.
Since his departure, Lancaster has been an adviser to the Football Association, the NFL and British Cycling.
He is interested in the Toulon job, but is understood to still be keen on a role in the southern hemisphere, with posts in Australia at the Queensland Reds and the Western Force both available.
Ground Truth: Former England boss Stuart Lancaster is meeting Toulon president Mourad Boudjellal this week as he seeks a return to full-time coaching.
C O NT:
Former England coach Stuart Lancaster is interested in a role in the southern hemisphere, according to the Football Association.
MLE: England’s Chris Lancaster has been appointed as the new head coach of the Toulon Football Association.
[Table]
in a
[Table]
Damage to to overhead wires meant the line is blocked north of Morpeth.
Virgin East Coast, Northern Rail, and Cross Country services were affected, with reports of large queues at Newcastle Central Station.
Buses were organised to take passengers between Newcastle and Edinburgh, with people advised to avoid travelling if possible.
Services resumed late on Friday
[Table]
Rail passengers travelling between Newcastle and Scotland faced
severe dis- ruption on Friday
[Table]
Rail services between Newcastle and
Edinburgh
have
been
disrupted after a power cut led to delays.
MLE
[Table]
Trains across the UK have been cancelled due to a disruption to the main line in the Highlands.
Document: Rashan Charles, 20, was wrestled to the ground in Dalston, east London, on 22 July, and died about an hour later.
On Friday, clashes broke out in Hackney as protesters blocked part of Kingsland Road and set mattresses alight.
A spokesman for Mr Charles’s family said they understood the anger but called for "dignified" protest. "
Burning down homes will not give justice," he said.
Mr Charles was pursued by officers and became ill after trying to swallow an object, the Met has said.
He died soon after in hospital.
The Independent Police Complaints Commission is investigating.
Police warned that anyone using Mr Charles’s death "as an excuse to commit crime" would be "dealt with robustly".
Appealing for calm, family spokesman Stafford Scott said: "We understand your frustration, we understand your anger - don’t feel that the family doesn’t feel the anger and the frustration too. "
But what the family knows is taking it to the streets doesn’t give you justice.
"Burning down your own homes, burning down your neighbourhood is not going to give you justice."
Mr Scott, who runs race advocacy group Tottenham Rights...
[Table]
The family of a black man who died after being apprehended by police has appealed for peace after violent protests in the wake of his death.
[Table]
The family of a man who died after he was attacked by anti-racism protesters have appealed for calm.
MLE: nll:
Thousands of black people have protested against the death of a man who was killed in a street attack.