D iffu COMET : Contextual Commonsense Knowledge Diffusion (2024)

Abstract: Inferring contextually-relevant and diverse commonsense to understand narratives remains challenging for knowledge models. In this work, we develop a series of knowledge models, DiffuCOMET, that leverage diffusion to learn to reconstruct the implicit semantic connections between narrative contexts and relevant commonsense knowledge. Across multiple diffusion steps, our method progressively refines a representation of commonsense facts that is anchored to a narrative, producing contextually-relevant and diverse commonsense inferences for an input context. To evaluate DiffuCOMET, we introduce new metrics for commonsense inference that more closely measure knowledge diversity and contextual relevance. Our results on two different benchmarks, ComFact and WebNLG+, show that knowledge generated by DiffuCOMET achieves a better trade-off between commonsense diversity, contextual relevance and alignment to known gold references, compared to baseline knowledge models.

## introduction identifying the commonsense inferences that underlie narratives, such as stories or dialogues @xcite , is crucial to understanding those same narratives. for example, to understand why "hank ... got the shopping bags" in the context in figure 1 , a model would need to infer that (1) hank was not finished wrapping gifts, and so (2) would need to buy more wrapping paper. however, comprehensively inferring these diverse, yet implicit, commonsense inferences that are relevant to a context remains a challenging task. recent methods for identifying contextuallyrelevant commonsense inferences (bosselut et al., † corresponding author. 1 we release our code to the community at https:// github.com/silin159/diffucomet 2021; @xcite use knowledge models @xcite to generate commonsense facts. while knowledge models have been less brittle than previous retrieval-based methods for commonsense inference, they have two major shortcomings. first, they are trained to verbalize tuples from general commonsense knowledge graphs @xcite , leading them to produce valid, but often contextually-irrelevant, commonsense inferences when applied out-of-the-box to real narratives. second, because they are trained using autoregressive training objectives, they subsequently decode high-likelihood, non-diverse sequences that only identify limited collections of commonsense inferences relevant to an input context. in this work, we address these challenges of contextual commonsense knowledge generation by developing diffusion @xcite commonsense transformer @xcite models. diffucomet models (shown in figure 1 ) uses diffusion-based decoding to generate relevant knowledge embeddings that are constrained to the narrative context. over multiple iterations of constrained diffusion, our models refine a latent representation of the semantic connections between a context and its relevant facts, ensuring that it generates commonsense knowledge that is more contextually relevant to the narrative. at the same time, by jointly refining multiple fact embeddings during diffusion, diffucomet also generates more diverse inferences than comparable-size autoregressive knowledge models. we evaluate diffucomet models using traditional nlg metrics (e.g., bleu; @xcite commonly used for evaluating knowledge models. however, these metrics focus on surface form matching to gold references, and fall short of measuring the diversity of commonsense inferences and their semantic relevance to real narrative contexts. our second contribution is a novel set of metrics that assess the diversity and contextual relevance of knowledge generated by knowledge models. using both the traditional evaluation metrics and our new suite, we evaluate our models on a commonsense inference linking benchmark @xcite ) that covers both social and physical knowledge, and a second knowledge generation benchmark that involves extracting rdf triplets from language, webnlg+ @xcite . our result show that diffucomet models generate knowledge that achieves a better balance of diversity and contextual relevance compared to other knowledge models. diffucomet models also more robustly generalize to generate knowledge for out-of-distribution narratives, and are better at producing novel knowledge tuples that are not in their initial training set. finally, on our second benchmark, webnlg+, we verify that our diffusion modeling method also generalizes well to a completely new factual knowledge generation task beyond the commonsense domain. ## background: diffusion models diffusion models learn to construct synthetic data from random noise. they use a forward process to gradually corrupt real data samples with additive noise, and learn a reverse process to recover (or de-noise) the corrupted data samples. through the de-noising of corrupted data, diffusion models learn to map from a random noise distribution to their target data distribution, which grounds their synthetic data generation. in this paper, we adopt the ddpm foot_0 @xcite formulation of the forward and reverse diffusion processes. specifically, based on a sample z 0 from a continuous input data distribution q(z 0 ), the forward process constructs noisy sample z t over a sequence of time steps @xmath0, 2, ..., t }. in ddpm, z t is sampled from a gaussian distribution conditioned on the previous sample z t-1 , given by: q(z t |z t-1 ) = n (z t ; 1β t z t-1 , β t i) in the reverse process, diffusion models learn an inverse distribution q(z t-1 |z t ) to de-noise samples created by the forward process. to more precisely couple the intermediate states of the reverse process with the final de-noised sample z 0 , diffusion-lm @xcite reformulates the task of predicting z t-1 as directly predicting z 0 (based on z t ), and uses a mean-squared error training loss on the z 0 prediction at each time step foot_1 : ## contextual knowledge diffusion in this section, we first introduce the task of contextual commonsense knowledge generation, and then propose diffucomet, our diffusion approach for this task. the overview of our method is presented in figure 1 . task description given a narrative sample s as context, e.g., a story snippet or a dialogue, the model needs to generate commonsense inferences as a set facts @xmath1, ..., k n , ..., k n }, which are relevant for understanding the situation described in the context. each fact k @xmath2, r n , a n ) is represented as a triple containing a head entity h n , a tail (attribute) entity a n , and a relation r n connecting them, e.g., (wrapping paper, used for, wrap gifts), as shown in figure 1 . we denote the set of unique head entities, relations and tail entities in k as h, r and a, respectively. ## evaluation prior work in commonsense knowledge generation @xcite evaluated knowledge models using traditional nlg metrics (e.g., bleu; @xcite in controlled studies with kgs, where the inputs to the models were head entities and relations and the knowledge model produced tail attributes. in practice, however, knowledge models are used to generate implicit commonsense inferences for natural language contexts @xcite , requiring generated inferences to be relevant to a more complex input than a basic kg head entity, and necessitating diverse generated inferences that comprehensively augment the context. however, traditional nlg metrics fall short of measuring these important dimensions because they measure surface form overlap between model outputs and references, which rewards generating facts with similar or duplicated semantics, limiting diversity. motivated by these shortcomings, we propose novel evaluation metrics that assess the diversity and contextual relevance of generated knowledge. first, to eliminate the effect of knowledge repetition in generations, we cluster similar facts and treat each fact cluster (instead of each single fact) as a unit piece of knowledge. in particular, we use the dbscan @xcite algorithm to group gold facts @xmath3, k 2 , ..., k n } and generated facts @xmath4, k2 , ..., k n } into clusters @xmath5, c 2 , ..., c m } and ĉ = {ĉ 1 , ĉ2 , ..., ĉ m }, respectively. we test two methods for measuring the similarity of facts for clustering: word-level edit distance @xcite , which measures the difference of two facts' surface-form tokens, and euclidean distance of sentence-bert @xcite embeddings, which measures the semantic difference of two facts. based on these clusters, we develop three metrics to measure the diversity of generated facts, their contextual relevance, and their alignment to gold references, as shown in figure 3 . ## experimental settings datasets first, we evaluate our approach on the comfact @xcite ) benchmark, where models need to generate atomic 20 20 @xcite social commonsense facts that are relevant to narrative contexts sampled from four diverse corpora: persona-chat @xcite , mu-tual @xcite , rocstories @xcite and cmu moviesummaries @xcite . we only use training data from the rocstories portion of comfact, to enable the evaluation of zero-shot generalization on the other three partitions of the dataset. our fact embedding module is pretrained on the full atomic 20 20 knowledge base, which contains ∼ 972k commonsense facts after preprocessing. 8 we also evaluate our approach in a conceptually different setting, the webnlg+ 2020 (ferreira et al., 2020) dataset, which consists of rdf @xcite facts sampled from the dbpedia @xcite knowledge base, with corresponding natural language texts verbalizations. the task is to generate the sampled rdf facts given their verbalizations. we use ∼13k facts from the training data to pretrain the fact embedding module. baselines we train diffucomet using bartbase and bart-large as pretrained models, and compare with three baselines developed on the same backbones: a) a greedy baseline that is trained to autoregressively generate the concatenation of all relevant facts, 9 b) a sampling baseline that uses nucleus sampling @xcite to generate multiple individual facts in parallel, and c) a diverse beam search baseline that uses diverse beam search to generate multiple inferences in parallel. we also compare our models trained using bart-large to baselines developed on models of similar scale: d) the aforementioned greedy decoding, sampling and beam search baselines trained from comet-bart @xcite , a bart-large model further pre-trained on atomic 20 20 for commonsense knowledge completion, and e) grapher @xcite , which trains a t5-large @xcite model to generate entities (nodes) related to the context, followed by a mlp classifier to predict the relations (edges) between entities. metrics we evaluate these methods on our clustering-based metrics described in section 4. as the clustering algorithm (i.e., dbscan) used in our metrics has an adjustable clustering granularity controlled by a distance threshold, we consider a range of distance thresholds and take the average of evaluation results across all thresholds in the range, allowing us to avoid biasing our metrics to a specific distance threshold. 10 for comfact, we also test on the metrics from @xcite for evaluating commonsense knowledge generation, including bleu @xcite , meteor @xcite and rouge-l @xcite . for evaluation on webnlg+ 2020, we also report the official metrics for this dataset's challenge @xcite , which construct 9 facts are concatenated by a special @xmath6. 10 we include more details of our clustering threshold selection in appendix c.2. optimal pairings between predicted facts and gold references, and then compute precision, recall, and f1 scores based on the surface-form matching of paired facts. we denote these webnlg metrics as web-prec., web-rec. and web-f1. foot_5 ## results and analysis for evaluation on the traditional nlg metrics, we find that diffucomet models score higher overall than most baseline models on metrics that check the alignment with gold references, i.e., bleu, meteor and rouge-l, except for the greedy decoding baseline, whose higher scores are artificially high because it generates very little knowledge, i.e., only ∼2 facts per context. we also include further comparisons of models with bart-base backbones in appendix f.1, where our models outperform baselines by a larger gap, i.e., ∼15% absolute ra-f1 improvement on average. we also test diffucomet's ability to generalize to out-of-domain contexts using the other portions of comfact with contexts sampled from personachat, mutual and moviesummaries. we report main generalization results to the above three portions in the results of our automatic evaluation are also supported by our human evaluation. we hire amazon mechanical turk workers 14 to evaluate the validity and contextual relevance of models' generated knowledge on the rocstories portion of comfact. specifically, given a narrative context 13 full evaluation results on out-of-domain contexts are reported in appendix f.2. and a list of commonsense facts that a model generates about the context, we ask three workers to independently judge whether each fact is valid (i.e., plausible and natural-sounding) and relevant foot_7 to the context, and take their majority vote as the assessment. in novelty diffucomet models also produce more novel commonsense inferences. a historical advantage of knowledge models (e.g., comet) was their ability to generate knowledge beyond the graphs they used for pretraining @xcite , making them powerful tools to generate commonsense knowledge for unseen narratives. to test the novelty of generated commonsense knowledge from diffucomet, we develop a heuristic method that identifies knowledge as novel if its maximum pair-wise (sentence-bert embedding) cosine similarity to comfact gold references is lower than 0.45. however, as this cut-off would likely cause invalid and irrelevant facts to be considered novel, we only include facts whose relevance score is higher than 0.97. 16 in ## related work commonsense knowledge grounding to augment nlp systems with commonsense knowledge, various systems for question answering @xcite @xcite and narrative generation @xcite use retrieval methods based on heuristics to link relevant facts from commonsense knowledge graphs @xcite @xcite . however, these systems typically have low precision when adapted to more general and complex commonsense linking @xcite . @xcite developed commonsense fact linking to improve retrieval precision, but this requires inefficiently traversing all candidate facts to check their contextual relevance. due to above limitations of retrieval-based knowledge grounding, one line of research @xcite uses knowledge models @xcite to generate tail inferences from narrative statements. however, these methods often produce irrelevant facts as the knowledge models are pre-trained for context-free knowledge graph completion. finally, developing new knowledge models to learn contextual commonsense generation turns out to be a promising track of research, while current works are limited to simple physical @xcite or rdf-style factual @xcite knowledge. we build new models to address contextual commonsense generation in a more general scope. diffusion models considerable recent works @xcite @xcite have developed methods to improve text generation with diffusion models @xcite @xcite . however, the potential of diffusion models in textto-knowledge generation is still under-explored. in this paper, we introduce diffusion models for the task of contextual knowledge generation. ## conclusion in this work, we leverage the power of diffusion models for contextual commonsense knowledge generation, and formulate novel metrics to highlight important dimensions of diversity and contextual relevance for this task. our diffusion knowledge models, diffucomet, outperform various autoregressive knowledge models, producing more diverse, novel, and contextually-relevant com-monsense knowledge, and achieving better out-ofdistribution performance. finally, our analysis reveals how diffucomet refines implicit knowledge representations over the course of the diffusion process to produce more relevant and diverse inferences, hinting at our method's potential benefit in other text-to-graph generation tasks.