Rule or Story, Which is a Better Commonsense Expression for Talking with Large Language Models? (2024)

Abstract: Building machines with commonsense has been a longstanding challenge in NLP due to the reporting bias of commonsense rules and the exposure bias of rule-based commonsense reasoning. In contrast, humans convey and pass down commonsense implicitly through stories. This paper investigates the inherent commonsense ability of large language models (LLMs) expressed through storytelling. We systematically investigate and compare stories and rules for retrieving and leveraging commonsense in LLMs. Experimental results on 28 commonsense QA datasets show that stories outperform rules as the expression for retrieving commonsense from LLMs, exhibiting higher generation confidence and commonsense accuracy. Moreover, stories are the more effective commonsense expression for answering questions regarding daily events, while rules are more effective for scientific questions. This aligns with the reporting bias of commonsense in text corpora. We further show that the correctness and relevance of commonsense stories can be further improved via iterative self-supervised fine-tuning. These findings emphasize the importance of using appropriate language to express, retrieve, and leverage commonsense for LLMs, highlighting a promising direction for better exploiting their commonsense abilities.

## introduction building machines with commonsense has been a longstanding goal in ai and nlp @xcite . despite advancements in large language models (llms), incorporating commonsense knowledge in these models remains a significant challenge @xcite @xcite , due to the reporting bias of commonsense knowledge and the exposure bias of commonsense reasoning @xcite . ## commonsense retrieval from llms as stories and rules in this section, we answer the first question: which expression, story or rule, is more effective for retrieving commonsense from the memory of llms? first, we compare the confidence in generating stories and commonsense rules. then, we employ an automatic evaluation to assess the accuracy of commonsense within the generated stories and rules. ## leveraging commonsense in stories and rules for problem solving this section answers the second question: which expression, story or rule, is more suitable for llms to leverage commonsense in solving problems? we compare the confidence of reasoning with stories or rules as contexts. then, we assess the performance of commonsense qa by employing either stories or rules as contexts and perform detailed analyses. ## iterative self-supervised fine-tuning after identifying the two issues in generating stories of past experiences, this section presents our iterative self-sft approach to address these issues. ## conclusion this paper systematically compares stories and rules as commonsense expressions to retrieve and leverage commonsense knowledge in llms. experimental results show that the story is a better expression for retrieving commonsense from llms. llms generate stories with more confidence and higher commonsense accuracy than rules. moreover, the story is a more effective commonsense expression for answering questions regarding daily events, while the rule is more effective for scientific commonsense qa. this phenomenon aligns with the reporting bias of commonsense. stories and rules can complement each other to further enhance answer accuracy. we provide further insights through in-depth analyses, highlighting two challenges in generated stories: commonsense hallucination and semantic drifting. we show that the correctness and relevance of commonsense stories can be improved via iterative self-supervised fine-tuning, underscoring the potential for selfimprovement in the commonsense ability of llms. this paper suggests a new perspective, going beyond the common practice of expressing commonsense as rules. our results and findings emphasize the importance of using the appropriate language to express, retrieve, and leverage commonsense for llms to further exploit their potential. the full extent of llms in handling commonsense is not yet fully realized, calling for future research to refine and improve the commonsense abilities of llms.