KFCN et: Knowledge Filtering and Contrastive Learning for Generative Commonsense Reasoning (2021)

Abstract: Pre-trained language models have led to substantial gains over a broad range of natural language processing (NLP) tasks, but have been shown to have limitations for natural language generation tasks with high-quality requirements on the output, such as commonsense generation and ad keyword generation. In this work, we present a novel Knowledge Filtering and Contrastive learning Network (KFCNet) which references external knowledge and achieves better generation performance. Specifically, we propose a BERT-based filter model to remove low-quality candidates, and apply contrastive learning separately to each of the encoder and decoder, within a general encoder–decoder architecture. The encoder contrastive module helps to capture global target semantics during encoding, and the decoder contrastive module enhances the utility of retrieved prototypes while learning general features. Extensive experiments on the CommonGen benchmark show that our model outperforms the previous state of the art by a large margin: +6.6 points (42.5 vs. 35.9) for BLEU-4, +3.7 points (33.3 vs. 29.6) for SPICE, and +1.3 points (18.3 vs. 17.0) for CIDEr. We further verify the effectiveness of the proposed contrastive module on ad keyword generation, and show that our model has potential commercial value.

## introduction pre-trained language models have achieved impressive results across a wide range of nlp tasks @xcite @xcite @xcite @xcite . however, their ability to accurately reflect factual knowledge or perform logical inference is still limited. to investigate the ability of systems to capture commonsense knowledge, datasets such as commonsenseqa @xcite , swag @xcite , and winogrande @xcite have been proposed. separate to these discriminative tasks that require models to choose the correct option from multiple candidates, commongen @xcite is framed as a generation task, and requires the system to generate a logical and coherent sentence describing an everyday scenario based on a concept set. experiments show that state-of-theart generation models are not adequate or accurate enough to generate plausible sentences or reflect commonsense assumptions in this setting. external knowledge provides not only information about the sorts of relationships that hold between concepts, to potentially guide generation models in capturing the implicit logic between concepts, but also interpretability. inspired by @xcite and @xcite , we adopt a retrieval-and-generation framework, and propose a bert-filter and two contrastive learning modules for retrieval and generation, respectively. for retrieval, previous research @xcite has shown that traditional sparse vector space models, such as tf-idf and bm25, perform better than dense representation-based retrieval on heavily entity-centric tasks such as fever @xcite . however, while using sparse vector space retrieval models can retrieve relevant prototypes that contain a set of concepts, there can be significant domain mismatches between the retrieved results and target distribution, making it difficult for generation models to bridge between prototypes and targets. we argue that a two-stage retrieval strategy alleviates this issue by combining sparse vector space search and dense representation filters. first, a sparse vector retrieval model is used to find passage candidates with high coverage of concept words, and then a dense vector-based filter is applied to score the candidates, and filter out low-quality prototypes. for generation, we apply contrastive learning to each of the encoder and decoder, in a general encoder-decoder architecture. the core idea of contrastive learning is to construct positive and negative samples from an anchor sample, and draw together the anchor and positive samples while pushing away the anchor from all negative samples in the embedding space during training. given that high-quality prototypes can be used as clusters of positive samples, we propose a decoder contrastive module that minimizes the distance between decoded sentence representations with distinct prototypes retrieved from the same concept set. common scenario information and abstract concept relationships can be learned based on the contrasts between different prototypes. moreover, we propose an encoder contrastive module to force the encoder to learn sentence representations, and save it into a global token which is visible to the decoder during decoding. in this way, global sentence-level semantics can be captured better. the main contributions of this work are threefold: (1) we demonstrate that adding a high-quality matching model to the word overlap-based retriever benefits entity-centric retrieval tasks; (2) we propose two contrastive learning modules that can be applied to a general encoder-decoder generation model; and (3) we conduct experiments on com-mongen and an ad keyword generation task, and show that our method achieves large-scale improvements on both tasks. ## method in this section, we detail our method: knowledge filtering and contrastive learning network (kfc-net). first, we introduce our prototype retrieval strategy together with the knowledge filter model. then we present our generative model, based on an encoder-decoder architecture with two contrastive learning modules. finally, we show how we adapt the moco contrastive learning framework, and deal with multiple positive samples. ## results ## conclusion in this paper, we present kfcnet: a novel knowledge filtering and contrastive learning model for retrieval-augmented sequence generation, which achieves state-of-the-art results on the common-gen benchmark. two contrastive learning modules are proposed to capture global target semantics and learn general features from multiple retrieved prototypes. a prototype retrieval ablation study showed the effectiveness of the proposed filter for filtering low-quality candidates, and further experiments on ad keyword generation showed that our model has potential commercial value. in the future, we plan to extend the contrastive module to more general settings, such as natural language understanding and representation learning.