Guided Knowledge Generation with Language Models for Commonsense Reasoning (2024)

Abstract: Large Language Models (LLMs) have achieved notable success in commonsense reasoning tasks, benefiting from their extensive world knowledge acquired through extensive pretraining. While approaches like Chain-of-Thought (CoT) have shown promise in enhancing LLMs’ reasoning capabilities, mitigating the influence of inaccurate commonsense knowledge remains a challenge, particularly for small-scale LLMs (e.g., those with less than 10B parameters). In this work, we propose a novel method named Guided Knowledge Generation (GuideKG) to address these issues. It presents three advantages: (i) Employing LLMs to generate knowledge explanations and to automatically assign labels based on the probability of correct answers eliminates the need for costly manual annotation in subsequent training. (ii) Training a new module called the ‘Know-Filter’, which is used to evaluate knowledge, and we have introduced a new loss to enhance its performance. (iii) Evaluating the effectiveness of knowledge fragments at the sentence level and fusing them allows for precise control over the generation process of LLMs. We evaluate our GuideKG on small-scale LLMs and show that it outperforms all baselines on four widely-used commonsense reasoning benchmarks. Moreover, our experiments reveal that, with proper guidance, small-scale LLMs can exhibit exceptional performance in commonsense reasoning.

## introduction commonsense reasoning abilities are crucial for achieving human-like intelligent systems, which encompass a comprehensive understanding of everyday world knowledge and the inference capacity to facilitate problem-solving and decision-making 𝑭𝟏: the concept of a month is based on the cycle of the moon, which takes approximately 29 days, and a calendar month is typically defined as a time period of 30 or 31 days, with the first day being the first of the month and the last day being the last day of the month. utility score: 0.45 𝒌𝟐 ′ : however, in some cases, a month may be defined as a shorter period of time, such as in leap years, where february has 29 days. 𝑭𝟏 + 𝒌2 ′ : utility score: 0.52 𝒌𝟑 ′ : a calendar month that has exactly 30 days is considered a short month, while a calendar month that has exactly 31 days is considered a long month. 𝑭𝟏 + 𝒌3 ′ : utility score: 0.47 𝒌𝟏 ′ : the leap year is a year that is a multiple of 4, but not a multiple of 100, and it has 366 days instead of 365 days. 𝑭𝟏 + 𝒌1 ′ : utility score: 0.22 𝑭𝟐: a month is considered long if it has 31 days, but in some cases, like leap years, a month can be shorter, such as february with 29 days. 𝑭𝟏 + 𝑭𝟐 : utility score: 0.67 𝑵𝒐 (✓; utility score: 0.67) a calendar month is typically defined as a period of time that spans 30 days or more, but it can also include an additional day to make it 31 days long. this additional day, known as a "leap day," occurs every fourth year and is used to keep the calendar in sync with the …. utility score: 0.42 ## experimental setup in this section, we will delineate the specific implementation details of our approach and explicate how it can be applied to various types of tasks. ## experimental results in this section, we present the performance of guidekg on commonsense reasoning and conduct a comparison with baseline models. additionally, detailed ablation studies were carried out. unless specifically stated otherwise, we sampled 10 knowledge sentences in each generation stage, with n set to 2 in the fusion mechanism. it can be observed that the enhancement in commonsense reasoning through sc is limited, aligning with findings from previous works on cot @xcite . when faced with commonsense questions, llms prefer producing correct answers based on accurate knowledge rather than just detailed steps. llms' chains of thought may include factual inaccuracies or misleading information, which could hinder fact-based commonsense reasoning @xcite . ## analysis we conduct theoretical experiments and case studies to investigate the performance ceiling and current performance of guidekg. ## conclusion we introduce guidekg, a cost-effective and effective framework to enhance the commonsense reasoning performance of small-scale llms. initially, we automatically collect training data and apply a novel utility-weighted classification loss to train a reliable know-filter. subsequently, by integrating sentence-level generation and fusion strategies, guidekg achieves significant improvements in commonsense reasoning benchmarks and demonstrates its efficacy across various datasets and models. we also showcase the impressive potential for improvements in commonsense reasoning tasks using small-scale llms.