Keywords and Instances: A Hierarchical Contrastive Learning Framework Unifying Hybrid Granularities for Text Generation (2022)

Abstract: Contrastive learning has achieved impressive success in generation tasks to militate the “exposure bias” problem and discriminatively exploit the different quality of references. Existing works mostly focus on contrastive learning on the instance-level without discriminating the contribution of each word, while keywords are the gist of the text and dominant the constrained mapping relationships. Hence, in this work, we propose a hierarchical contrastive learning mechanism, which can unify hybrid granularities semantic meaning in the input text. Concretely, we first propose a keyword graph via contrastive correlations of positive-negative pairs to iteratively polish the keyword representations. Then, we construct intra-contrasts within instance-level and keyword-level, where we assume words are sampled nodes from a sentence distribution. Finally, to bridge the gap between independent contrast levels and tackle the common contrast vanishing problem, we propose an inter-contrast mechanism that measures the discrepancy between contrastive keyword nodes respectively to the instance distribution. Experiments demonstrate that our model outperforms competitive baselines on paraphrasing, dialogue generation, and storytelling tasks.

## introduction generation tasks such as storytelling, paraphrasing, and dialogue generation aim at learning a certain correlation between text pairs that maps an arbitrary-length input to another arbitrary-length output. traditional methods are mostly trained with "teacher forcing" and lead to an "exposure bias" problem @xcite . incorporating the generation method with contrastive learning achieved impressive performance on tackling such issues, which takes an extra consideration of synthetic negative samples contrastively @xcite . figure 1 : the semantic meaning of the sentence "what are the best books on cosmology?" would be greatly changed if the keyword "cosmology" is changed to "astrophysic". existing contrastive mechanisms are mainly focused on the instance level @xcite . however, word-level information is also of great importance. take the case shown in the upper part of figure 1 for example, the keyword covers the gist of the input text and determines the embedding space of the text. the text representation will be significantly affected if adding a slight perturbation on the keyword, i.e., changing "cosmology" to "astrophysics". in addition, as shown on the bottom part, under some circumstances, it is too easy for the model to do the classification since the semantic gap between contrastive pairs is huge. thus, the model fails to distinguish the actual discrepancy, which causes a "contrast vanishing" problem at both instance-level and keyword-level. based on the above motivation, in this paper, we propose a hierarchical contrastive learning method built on top of the classic cvae structure. we choose cvae due to its ability in modeling global properties such as syntactic, semantic, and discourse coherence @xcite . we first learn different granularity representations through two independent contrast, i.e., instancelevel and keyword-level. specifically, we use the universal and classic textrank @xcite ) method to extract keywords from each text, which contain the most important information and need to be highlighted. on the instancelevel, we treat the keyword in the input text as an additional condition for a better prior semantic distribution. then, we utilize kullback-leibler divergence @xcite to reduce the distance between prior distribution and positive posterior distribution, and increase the distance with the negative posterior distribution. while on the keyword-level, we propose a keyword graph via contrastive correlations of positive-negative pairs to learn informative and accurate keyword representations. by treating the keyword in the output text as an anchor, the imposter keyword is produced by neighboring nodes of the anchor keyword and forms the keyword-level contrast, where the similarity between the imposter keyword and the anchor keyword is poorer than the positive keyword. to unify individual intra-contrasts and tackle the "contrast vanishing" problem in independent contrastive granularities, we leverage an inter-contrast, the mahalanobis contrast, to investigate the contrastive enhancement based on the mahalanobis distance @xcite , a measure of the distance between a point and a distribution, between the instance distribution and the keyword representation. concretely, we ensure the distance from the anchor instance distribution to the groundtruth keyword vector is closer than to the imposter keyword vector. the mahalanobis contrast plays an intermediate role that joins the different granularities contrast via incorporating the distribution of instance with the representation of its crucial part, and makes up a more comprehensive keyworddriven hierarchical contrastive mechanism, so as to ameliorate the generated results. we empirically show that our model outperforms cvae and other baselines significantly on three generation tasks: paraphrasing, dialogue genera-tion, and storytelling. our contributions can be summarized as follows: ## conclusion in this paper, we propose a hierarchical contrastive learning mechanism, which consists of intra-contrasts within instance-level and keywordlevel and inter-contrast with mahalanobis contrast. the experimental results yield significant out-performance over baselines when applied in the cvae framework. in the future, we aim to extend the contrastive learning mechanism to different basic models, and will explore contrastive learning methods based on external knowledge.