C om F act: A Benchmark for Linking Contextual Commonsense Knowledge (2022)

Abstract: Understanding rich narratives, such as dialogues and stories, often requires natural language processing systems to access relevant knowledge from commonsense knowledge graphs. However, these systems typically retrieve facts from KGs using simple heuristics that disregard the complex challenges of identifying situationally-relevant commonsense knowledge (e.g., contextualization, implicitness, ambiguity).In this work, we propose the new task of commonsense fact linking, where models are given contexts and trained to identify situationally-relevant commonsense knowledge from KGs. Our novel benchmark, ComFact, contains ~293k in-context relevance annotations for commonsense triplets across four stylistically diverse dialogue and storytelling datasets. Experimental results confirm that heuristic fact linking approaches are imprecise knowledge extractors. Learned fact linking models demonstrate across-the-board performance improvements (~34.6% F1) over these heuristics. Furthermore, improved knowledge retrieval yielded average downstream improvements of 9.8% for a dialogue response generation task. However, fact linking models still significantly underperform humans, suggesting our benchmark is a promising testbed for research in commonsense augmentation of NLP systems.

## introduction in conversations, stories, and other varieties of narratives, language users systematically elide information that readers (or listeners) reliably fill in with world knowledge. for example, in figure 1 , the speaker of utterance t (i.e., pink) infers that their counterpart (cyan) wants to be a doctor because they are studying medicine, even though the cyan speaker does not explicitly mention their career † corresponding author. 1 we release our data and code to the community at https: //github.com/silin159/comfact utterance t-1: i continue to write while studying medicine. utterance t: good luck, being a doctor is hard. maybe you will write medical books. utterance t+1: not a chance! i love making up stories. medicine is too real sometimes. book, used for, learning about medicine x wants to be a doctor, but before, x needs, go to college x writes books, because x wants, to tell stories good, used for, destroying evil goals. to reflect this ability, language understanding systems are often augmented with knowledge bases (kbs, e.g., @xcite that allow them to access relevant background knowledge. considerable research has examined how to construct large databases of world knowledge for this purpose @xcite @xcite , as well as how to design models that can reason over relevant subsets of this knowledge to form a richer understanding of language (e.g., @xcite . however, less work examines how to retrieve these inferences (or facts) from the kb in the first place. current methods typically rely on pattern-based heuristics @xcite , unsupervised scoring using corpus statistics @xcite or neural re-rankers @xcite , or combinations of these methods @xcite . these simple methods produce computationally tractable knowledge representations, but frequently retrieve noisy information that is irrelevant to the narrative they are constructed to represent. re-cent work demonstrates that models trained with heuristically-retrieved commonsense knowledge learn simplified reasoning patterns @xcite and provide false notions of interpretability @xcite . we posit that inadequate retrieval from large-scale knowledge resources is a key contributor to the spurious reasoning abilities learned by these systems. acknowledging the importance of retrieving relevant commonsense knowledge to augment models, we identify a set of challenges that commonsense knowledge retrievers must address. first, retrieved commonsense knowledge must be contextuallyrelevant, rather than generically related to the entities mentioned in the context. second, relevant commonsense knowledge can often be implicit, e.g., in figure 1 , writing may be a leisure hobby for the cyan speaker, explaining why they "love making up stories". finally, knowledge may be ambiguously relevant to a context. the cyan speaker in figure 1 may write as a relaxing hobby, or be thinking of quitting medical school to pursue a career as a writer. without knowing the rest of the conversation, both inferences are potentially valid. to more adequately address these challenges, we introduce the new task of commonsense fact linking, 2 where models are given contexts and trained to identify situationally-relevant commonsense knowledge from kgs. for this task, we construct a commonsense fact linking dataset (comfact) to benchmark the next generation of models designed to improve commonsense fact retrieval. comfact contains ∼293k contextual relevance annotations for four diverse dialogue and storytelling corpora. our empirical analysis shows that heuristic methods over-retrieve many unrelated facts, yielding poor performance on the benchmark. meanwhile, models trained on our resource are much more precise extractors with an average 34.6% absolute f1 boost (though they still fall short of human performance). the knowledge retriever developed on our resource also brings an average 9.8% relative improvement on a downstream dialogue response generation task. these results demonstrate that comfact is a promising testbed for developing improved fact linkers that benefit downstream nlp applications. commonsense fact linking knowledgeintensive nlp tasks are often tackled using commonsense kgs to augment the input contexts provided by the dataset @xcite @xcite . models for various nlp applications benefit from this fact linking, including question answering @xcite @xcite , dialogue modeling @xcite and story generation @xcite . all above works typically conduct fact linking using heuristic solutions. recent research explores unsupervised learning approaches for improving on the shortcomings of heuristic commonsense fact linking. @xcite and @xcite use soft matching based on embedding similarity to link commonsense facts with implicit semantic relatedness. @xcite use knowledge-enhanced pretraining to implicitly incorporate commonsense facts into narrative systems, but their approach reduces the controllability and interpretability of knowledge integration. finally, several works @xcite @xcite use knowledge models @xcite @xcite to generate commonsense facts instead of linking from knowledge graphs. however, the contextual quality of generated facts from knowledge models is also under-explored in these application scenarios. in this paper, we conduct more rigorous study on commonsense fact linking. ## comfact construction in this section, we give an overview of commonsense fact linking and its associated challenges, and describe our approach for building the comfact dataset centered around these challenges. ## comfact analysis we use atomic 20 20 @xcite as the commonsense kg for building comfact, which contains 1.33m complex facts covering physical objects, daily events and social interactions. atomic 20 20 is a rich resource for building our dataset, as it covers rich knowledge types (e.g., physical, social, and event knowledge), and is partially consolidated from other popular kgs including conceptnet @xcite ) and atomic @xcite , potentially offering better generalization for fact linking with these other resources. as persona-atomic, mutual-atomic, roc-atomic and movie-atomic in comfact. 7 contextual relevance to quantitatively measure the ambiguity of linked commonsense facts' relevance, we use cohen's κ @xcite to measure the agreement between workers that annotate the same facts. most κ scores fall within the ranges that cohen described 7 see appendix a for more data collection details. as "moderate" (0.4 -0.6) or "substantial" (0.6 -0.8) agreement. we do observe that implicitly linked fact candidates have lower κ scores in their relevance validation, likely because they are linked to the context in a less straightforward way, which may lead to more subjective relevance judgements. link types ## experimental methods we evaluate our new benchmark using various baseline classification methods based on neural language models. all lms are individually trained and evaluated on the four comfact datasets. approach our models encode the concatenation of a narrative context with each of its fact candidates selected in sec 3.2. the output hidden states of the language models are then input to a binary classifier, which predicts whether the fact candidate is relevant to the context. we consider two context windows @xmath0, u t ] and models we use various pretrained language models as encoders for classifying facts, particularly the bert @xcite and roberta @xcite model families. we test base and large sizes of these models, as well as more light-weight distilbert @xcite and more advanced deberta @xcite . we also test the performance of a two-layer bi-directional lstm. we evaluate each model on: a) direct prediction, where the model is trained to directly classify the fact candidates, and b) pipeline prediction, where a first model is trained to classify head entities, and a second model classifies fact candidates associated with relevant head entities. 10 as a baseline, we report the performance of the same heuristic used to generate candidates, which predicts all fact candidates retrieved in sec 3.2 as relevant (i.e., the typical linking approach for many methods). we also include a semi-heuristic baseline head linking, which finetunes roberta (large) to only classify the head entities linked in sec 3.2, and predicts the relevance of each fact candidate as the relevance of its head. finally, we run a human study on a randomly sampled set of 200 contexts from each of the 4 test sets of comfact, and ask crowdworkers to judge the relevance of 3 linked facts with respect to each context. ## experimental results we report the accuracy and f1 of fact candidate classification on comfact in the heuristic baselines, showing supervised neural classification on top of heuristically selected facts can significantly improve fact linking quality. furthermore, the improvement over the head linking baseline demonstrates the importance of linking facts individually, rather than relying on coarsegrained entity linking. however, model-based fact linkers are still far from human performance on comfact, demonstrating that there is still considerable room for improvement. interestingly, we find that directly predicting fact links outperforms a pipeline approach that first predicts relevant head entities and then only classifies fact candidates of relevant head entities. 12 entity linking despite the error propagation, we see in ## conclusion in this work, we propose a general commonsense fact linking task that addresses the challenges of identifying relevant commonsense inferences for textual contexts: contextualization, implicitness, and ambiguity. to promote research into commonsense fact linking, we construct a new, challenging benchmark, comfact, of over 293k contextuallylinked commonsense facts. our experimental results show that the predominant heuristic methods used to select relevant commonsense facts for downstream applications perform poorly, motivating the need for new methods that can predict contextually-relevant commonsense inferences. however, commonsense inferences may connect concepts across longer time windows, which may affect how well models trained on our dataset scale to longer text snippets (as hinted by the lower transfer numbers when models trained on other data subsets are evaluated on movie-atomic). more broadly, due to the large number of knowledge graphs, narrative corpora, and pretrained models, this work cannot include an exhaustive coverage of this cross-section. instead, we identify a diverse subset of these resources for our study, though our datasets and methods all use english as a primary language. finally, some of the models in this work are based on large-scale language models, e.g., bert and roberta, which require considerable resources for inference. in the long run, if neural fact linkers are to be used for large numbers of data snippets (perhaps in online settings), models that can perform faster inference will be needed. zhi-xiu ye, qian chen, wen wang, and zhen-hua ling. 2019. align, mask and select: a simple method for incorporating commonsense knowledge into language representation models. arxiv preprint arxiv:1908.06725. da yin, li dong, hao cheng, xiaodong liu, kai-wei chang, furu wei, and jianfeng gao. 2022. a survey of knowledge-intensive nlp with pre-trained language models. arxiv preprint arxiv:2202.08772. saizheng zhang, emily dinan, jack urbanek, arthur szlam, douwe kiela, and jason weston. 2018. personalizing dialogue agents: i have a dog, do you have pets too? in proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pages 2204-2213. xikun zhang, antoine bosselut, michihiro yasunaga, hongyu ren, percy liang, christopher d. manning, and jure leskovec. 2022. greaselm: graph reasoning enhanced language models for question answering. in proceedings of the 10th international conference for learning representations (iclr). hao zhou, tom young, minlie huang, haizhou zhao, jingfang xu, and xiaoyan zhu. 2018. commonsense knowledge aware conversation generation with graph attention. in proceedings of the 27th international joint conference on artificial intelligence, pages 4623-4629. pei zhou, karthik gopalakrishnan, behnam hedayatnia, seokhwan kim, jay pujara, xiang ren, yang liu, and dilek hakkani-tur. 2022. think before you speak: explicitly generating implicit commonsense knowledge for response generation. in proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pages 1237-1252.