Coconut: Contextualized Commonsense Unified Transformers for Graph-Based Commonsense Augmentation of Language Models (2024)

Abstract: In this paper, we introduce COCONUT to effectively guide the contextualization of structured commonsense knowledge based on largelanguage models. COCONUT employs a contextualized knowledge prompting scheme to gather high-quality contextualization examplesfrom a large language model. These examples are subsequently distilled into small language models to enhance their contextualization capability. Extensive evaluations show that COCONUT considerably improves commonsense reasoning performance across diverse benchmarks, models, and settings, exhibiting its flexibility and universality in generating contextualized commonsense knowledge. Notably,COCONUT consistently outperforms the state-of-the-art technique by an average of 5.8%.

## introduction commonsense reasoning constitutes a significant challenge within natural language processing. while scaling language models using considerably more data and parameters has led to significant improvements in commonsense reasoning tasks @xcite , several studies have demonstrated that pre-trained language models possess a limited understanding of commonsense knowledge @xcite . these have triggered approaches to integrate external knowledge into language models to improve their commonsense reasoning abilities. to enhance the commonsense capability of language models, typical approaches draw commonsense knowledge from symbolic commonsense knowledge graphs (cskgs) @xcite , which are repositories encapsulating hand-crafted commonsense knowledge about objects, concepts, and events. these approaches augment language model representations 1 the code is available at https://github.com/ irishev/coconut ## contextualized knowledge prompting in this section, we present a contextualized knowledge prompting scheme, which generates contextualization examples using commonsense questions and cskgs. specifically, we prompt language models to generate examples of context-aware link and knowledge contextualization, as illustrated in figure 2 . ## coconut in this section, we introduce coconut, a framework designed to augment language models with contextualized commonsense knowledge. we present two coconut models: (1) coconut linker and (2) coconut generator. these two models generate contextualized knowledge from commonsense reasoning examples, as described in figure 3 . then, we describe the integration of the contextualized knowledge generated by the co-conut models. ## related work while large language models yield state-of-theart performance on many commonsense reasoning tasks, their pre-training objectives do not explicitly guide them to reason using commonsense knowledge @xcite , resulting in unsatisfactory performance in many real-world scenarios @xcite . to address the limitation, existing work has explored augmented language models to improve their commonsense reasoning ability. a typical approach is incorporating external knowledge from cskgs, thereby supplementing the limited textual information @xcite @xcite . another approach involves training a language model on commonsense corpora @xcite . recently, a line of research @xcite @xcite has proposed to generate knowledge by prompting language models due to the lack of scalability in utilizing cskgs. some recent methods have explored retrieving in-domain commonsense documents from a task-relevant corpus to improve commonsense reasoning capabilities @xcite @xcite . coconut provides two distinct advantages over existing commonsense augmentation methods. first, coconut alleviates the inherent limitation of cksgs, the lack of coverage, by introducing language models in contextualization. since language models have wide coverage and strong expressive power, they can effectively complement the coverage of cskgs. second, coconut utilizes cskgs as pivots in knowledge generation, which facilitates to generate a wide range of accurate knowledge and achieves generalization on small language models. ## conclusion in this paper, we have proposed coconut that contextualizes structured knowledge from cskgs, guided by large language models. our experimental results have verified that coconut outperforms state-of-the-art knowledge augmentation methods on diverse commonsense benchmarks. these show that large language models can explicitly guide the contextualization, leading to significant improvements in commonsense reasoning. furthermore, our analyses suggest that prompting with structured knowledge may be a promising approach to address hallucination in knowledge prompting.