N euro C omparatives: Neuro-Symbolic Distillation of Comparative Knowledge (2024)

Abstract: Comparative knowledge (e.g., steel is stronger and heavier than styrofoam) is an essential component of our world knowledge, yet understudied in prior literature. In this paper, we harvest the dramatic improvements in knowledge capabilities of language models into a large-scale comparative knowledge base. While the ease of acquisition of such comparative knowledge is much higher from extreme-scale models like GPT-4, compared to their considerably smaller and weaker counterparts such as GPT-2, not even the most powerful models are exempt from making errors. We thus ask: to what extent are models at different scales able to generate valid and diverse comparative knowledge?We introduce NeuroComparatives, a novel framework for comparative knowledge distillation overgenerated from language models such as GPT-variants and LLaMA, followed by stringent filtering of the generated knowledge. Our framework acquires comparative knowledge between everyday objects, producing a corpus of up to 8.8M comparisons over 1.74M entity pairs - 10X larger and 30% more diverse than existing resources. Moreover, human evaluations show that NeuroComparatives outperform existing resources in terms of validity (up to 32% absolute improvement). Our acquired NeuroComparatives leads to performance improvements on five downstream tasks.We find that neuro-symbolic manipulation of smaller models offers complementary benefits to the currently dominant practice of prompting extreme-scale language models for knowledge distillation.

## introduction in their book "surfaces and essences" on concepts and analogies, @xcite elucidate how concept learning requires comparing a pair of concepts, and parsing out their similarities and dissimilarities. indeed, comparative knowledge is an essential component of our world knowledge @xcite , underpinning some of the classical commonsense reasoning problems. for example, the problem "the large ball crashed right through the table because it was made of @xcite . what was made of @xcite ?" in winograd schema challenge @xcite requires comparing the relative strength between steel and styrofoam. yet, compared to general knowledge acquisition, there has been relatively little research focus on comparative knowledge acquisition, possibly due to the longstanding challenges of high-quality knowledge acquisition itself, let alone comparative knowledge. the few resources for comparative knowledge, all derived from web mining, however limited in size and diversity ( §5), have nonetheless been useful for challenging multimodal reasoning tasks @xcite , highlighting the value of comparative knowledge. in this paper, we draw inspirations from such literature about concept learning and inquire two related questions on comparative knowledge: (1) how well do models at different scales fare at the task of producing large-scale, high-quality comparative knowledge about a broad range of concepts? and (2) what are the implications for downstream tasks? compared to prior resources of commonsense knowledge acquired either via crowdsourcing @xcite or via information extraction (e.g., webchild @xcite and aser @xcite ), our attempt to (re-)focus on the task of comparative knowledge acquisition takes the perspective of "language models as knowledge bases" @xcite , motivated by the dramatic improvements in the capabilities of extreme-scale neural language models. collecting comparable entities overgenerating comparatives filtering comparatives retrieving wikidata entity sets blender food processor rice cooker pasta maker … expanding coverage categorybuilder blender food processor grinder microwave rice cooker pasta maker fryer coffee maker prensa ironing home keg-tapper … filtering obscure entities templating prompts compared to blenders, food processors … generating from proprietary llms controlled generation with open-source lms + neurologic are often simpler positive constraints have would typically often more softer negative constraints are always.. punctuations; pronouns; discourse connectives gpt-2 / llama gpt-3.5 / gpt-4 knowledge discrimination filtering contradictions deduplication filtering by constraints satisfied ## distilling neurocomparatives our framework for distilling comparatives from an autoregressive lm comprises three stages, illustrated in figure 1 . first, we collect comparable entities to construct prompts for eliciting comparative knowledge statements ( §2.1). next, we employ lms to overgenerate (potential) comparatives for every pair of selected entities ( §2.2). finally, we filter the generations ( §2.3) to obtain a large-scale, high-quality collection of comparative statements, which we call neurocomparatives (ncs) ( §3). ## neurocomparatives our large-scale generation effort produced 8.8m comparatives before discriminative filtering (details in app. a), which we refer to as nc-xl. specifically, nc-xl includes 8.7m comparatives generated from gpt-2 xl, 78k from llama-2-7b, 16.3k from instructgpt, 10.7k from chatgpt, and 6.6k from gpt-4. we further apply our knowledge discriminator model on nc-xl to create subsets containing only the top-50% and top-20% of comparatives, which we refer to as nc-l and nc-s, respectively. while nc-xl contains the greatest breadth of comparative knowledge, the more stringent filtering applied to produce nc-l and nc-s results in the highest knowledge quality, without hurting diversity. the largest existing resource of comparative commonsense knowledge is webchild @xcite , collected via information extraction methods. while webchild contains over 18m general assertions covering 2m concepts and activities, we focus on its comparative knowledge, which spans 813k assertions over 335k entity pairs. compared to webchild, our nc-xl corpus is 10x larger. the first set of examples for the entity pair (helicopters, planes) illustrates the more detailed, domain-specific properties, such as "operating costs", "more cargo", and "stable in flight" in nc-xl. in contrast, webchild assertions are more generic (e.g., "cooler", "better") and not specific to the domain of flight. this example also highlights how nc-xl assertions are more informative and interesting to humans, as evidenced by their lower rate of rejection shown in we also compare our generated knowledge to atomic @xcite and conceptnet @xcite . although neither explicitly contains comparative knowledge, they do contain relations from which comparatives can be inferred. we use the atlocation and madeupof relations in atomic, as well as the atlocation, partof, and madeof relations in conceptnet, to infer size comparisons over entities. we convert these entries to our ncs format for evaluation; for e.g., the atomic triple (human body, madeupof, brain) results in the comparative: "compared to brains, human bodies are larger.". we use human ( §3.1) and automatic ( §3.2) evaluation to compare the quality and diversity of neurocomparatives with existing comparative knowledge resources. ## downstream task performance while neurocomparatives are demonstratably diverse ( §3.2), the impact on downstream task can be measured via their coverage on existing benchmarks. here, we demonstrate ncs' wide knowledge coverage without compromising knowledge quality on three benchmarks. to this end, we use two existing benchmarks: elephant @xcite and verb physics reannotated @xcite . elephant contains 486 comparisons of sizes of various transportation vehicles and animals. vpr reannotates verbphysics @xcite by filtering out the examples where the objects were not comparable, or the annotators disagreed among themselves. this re-annotation results in 2964 examples of object relations (1927 unique pairs of objects) along the dimensions of speed, length and mass. the unstructured knowledge in neurocomparatives facilitates easy integration with llms through natural language, therefore enabling the use of comparative reasoning abilities of llms. as there are no benchmarks for systematically evaluating the comparative reasoning capabilities, we constructed a new comparative reasoning qa benchmark, comparativeqa to investigate this question. specifically, we identified all comparatives that were unanimously assigned a label of 'true' by annotators in our human evaluation of validity experiments ( §3.1). we then rephrased these comparatives as questions in order to construct a qa benchmark (see app. e.2). this produced a total of 656 comparative reasoning questions. finally, the copen dataset @xcite ) is a benchmark for probing conceptual knowledge in pre-trained language models. two tasks from copen (conceptual similarity judgement and conceptualization in contexts) are related to our task of acquiring comparative knowledge. therefore, we evaluate the benefit of our framework on these two additional downstream tasks. ## conclusion we demonstrate distillation of high-quality comparative knowledge from lms at different scales and produce neurocomparatives: the largest comparative knowledge corpus to date. nc-xl is 10x larger, 30% more diverse, and 19% more human acceptable than existing sources; with knowledge discrimination, we additionally achieve over 90% human acceptance. our work highlights the value of comparative knowledge and exploits both neurosymbolic manipulation of small-scale models and extreme-scale models for knowledge distillation. in our evaluations which demonstrated the utility of neurocomparatives for downstream tasks, we primarily focused on comparative reasoning since it is most directly related to the knowledge we acquired. however, a promising direction for future research would be investigating other downstream tasks which can benefit from training on neuro-comparatives.