CAR : Conceptualization-Augmented Reasoner for Zero-Shot Commonsense Question Answering (2023)

Abstract: The task of zero-shot commonsense question answering evaluates models on their capacity to reason about general scenarios beyond those presented in specific datasets. Existing approaches for tackling this task leverage external knowledge from CommonSense Knowledge Bases (CSKBs) by pre-training the model on synthetic QA pairs constructed from CSKBs. In these approaches, negative examples (distractors) are formulated by randomly sampling from CSKBs using fairly primitive keyword constraints. However, two bottlenecks limit these approaches: the inherent incompleteness of CSKBs limits the semantic coverage of synthetic QA pairs, and the lack of human annotations makes the sampled negative examples potentially uninformative and contradictory. To tackle these limitations above, we propose Conceptualization-Augmented Reasoner (CAR), a zero-shot commonsense question-answering framework that fully leverages the power of conceptualization. Specifically, CAR abstracts a commonsense knowledge triple to many higher-level instances, which increases the coverage of the CSKB and expands the ground-truth answer space, reducing the likelihood of selecting false negative distractors. Extensive experiments demonstrate that CAR more robustly generalizes to answering questions about zero-shot commonsense scenarios than existing methods, including large language models, such as GPT3.5 and ChatGPT. Our code, data, and model checkpoints are available at https://github.com/HKUST-KnowComp/CAR.

## introduction pre-trained language models (plms; @xcite fine-tuned on task-specific training sets achieve remarkable near-human performance on held-out test sets, yet struggle to generalize to examples that are distributionally different @xcite . the simple heuristic used in this process can result in false negative options. ## car framework this section introduces our proposed car framework. a general sketch is presented in figure 3 . our framework can be summarized into three steps: ## analysis and discussion in this section, we study the effects of conceptualization and the reasons contributing to car's success. first, we conduct expert evaluations on the synthetic qa pairs to study the quality and diversity of different cskb augmentation methods in comparison with conceptualization. second, we conduct training dynamics @xcite analysis to show that conceptualization-aided qa pairs can provide more ambiguous examples help-ful for training. finally, we study the impact of filtering atomic10x with different critic thresholds, the ablations of car, and the effect of conceptualization from an out-of-domain generalization perspective in appendix b.5, b.7, and b.8. ## conclusions in this paper, we present car, a pioneering framework for zero-shot commonsense qa empowered by conceptualization. our approach surpasses even large language models on five qa benchmarks, achieving state-of-the-art performance on average. our analyses reveal that conceptualization can improve the sampling of negative examples, and abstract knowledge is more helpful compared with those distilled from gpt3 as it provides more ambiguous knowledge to support ood generalization. these findings demonstrate the substantial benefits of introducing conceptualization and abstract knowledge into zero-shot commonsense reasoning.