In Search of the Long-Tail: Systematic Generation of Long-Tail Inferential Knowledge via Logical Rule Guided Search (2024)

Abstract: To effectively use large language models (LLMs) for real-world queries, it is imperative that they generalize to the long-tail distribution, i.e. rare examples where models exhibit low confidence. In this work, we take the first step towards evaluating LLMs in the long-tail distribution of inferential knowledge. We exemplify long-tail evaluation on the Natural Language Inference task. First, we introduce Logic-Induced-Knowledge-Search (LINK), a systematic long-tail data generation framework, to obtain factually-correct yet long-tail inferential statements. LINK uses variable-wise prompting grounded on symbolic rules to seek low-confidence statements while ensuring factual correctness. We then use LINK to curate Logic-Induced-Long-Tail (LINT), a large-scale long-tail inferential knowledge dataset that contains 108K statements spanning four domains. We evaluate popular LLMs on LINT; we find that state-of-the-art LLMs show significant performance drop (21% relative drop for GPT4) on long-tail data as compared to on head distribution data, and smaller models show even more generalization weakness. These results further underscore the necessity of long-tail evaluation in developing generalizable LLMs.

## introduction generalization, especially to unfamiliar and novel situations, is a cornerstone for the usability of large language models (llms) in addressing varied real-world inquiries. this imminent demand necessitates evaluation of llms in the longtail distribution (the space consisting of unfamiliar examples on which the model has low confidence). previous works, mostly focusing on model memorization issue, define long-tail knowledge using the frequency of entities in a knowledge base @xcite , in the pre-training dataset @xcite , or in wikipedia search @xcite . @xcite introduces a general definition for long-tail statements, where long-tail examples are assigned lower likelihood by a pre-trained language model. we follow this definition which applies to various data format and language task -for any set of statements with similar length and format, those in the long-tail distribution cannot be generated or are generated with low confidence by the models, compared to those in the head distribution. recent works have noticed that llms have a marked decline in performance when facing inputs from the long-tail @xcite ). hallucination, for example, is found to be correlated with data being in the long-tail distribution @xcite . llms' ineffective utilization of long-tail knowledge impacts its reasoning capabilities and raises reliability concerns in downstream tasks @xcite . evaluation in the long-tail distribution requires systematic generation of long-tail data. however, obtaining examples in the long-tail is non-trivial. with state-of-the-art llms being trained on vast volume of data on the internet @xcite , it is increasingly difficult to find unseen examples that can effectively test model generalization to its low-confidence end. crowdsourcing long-tail data is also difficult because of human cognitive bias @xcite , and llms' generation in the longtail distribution is hindered by their pretraining task of "most likely" next token @xcite . while demonstrating long-tail evaluation across all applications and domains is not feasible within the scope of one paper, this work focuses on inferential knowledge statement in the form of natural language inference (nli) task @xcite : nli requires extensive knowledge and complex reasoning about entities and events, and is one of such tasks on which llms llama2 chatgpt gpt4 long-tail head training data has long-tail distribution knowledge with low confidence especially requires model generalization. we propose long-tail evaluation have impressive performance @xcite @xcite . following @xcite , we structure inferential knowledge as if-then relations with variables, written in a premise, conclusion format ( first, we make long-tail inferential knowledge generation possible. we propose a novel and lightweight long-tail inferential knowledge generation framework, logic-induced-knowledge-search (link ) ( § 2), a variable-wise prompting framework grounded on symbolic rules. this framework enables us to obtain long-tail knowledge statements from existing llms. our evaluation shows that by taking simply instructions chatgpt(gpt-3.5-turbo) and gpt4(gpt-4) can only produce statements in the head distribution that also have lower factual correctness, but using link with the llms improves on both distribution conformity and factual correctness ( § 3). second, we test llms' long-tail generalization capability on data generated by link( § 4). we produce a large-scale dataset, logic-induced-long-tail (lint), which contains 108k knowledge statements spanning across 4 different domains( our work is the first to propose a systematic framework that generates data in the long-tail distribution. using nli as an example, we show that generating data in the long-tail distribution is an effective way for curating evaluation examples for llm generalization. our work serves as a starting point for the series of research on long-tail data discovery and generation, and motivates the community to incorporate long-tail evaluation into model building pipelines. search, our novel variable-wise search pipeline. ## generating long-tail inferential knowledge with link in this section, we compare link's ability to generate long-tail inferential knowledge with instructiononly llms, chatgpt and gpt4, who do not use knowledge beam search. ## llms' (lack of) generalization in the long-tail distribution using data from lint, we evaluate llm generalization through an entailment classification task on inferential knowledge in the long-tail distribution. we use all human evaluated knowledge statements except for those with incorrect data types in lint, with 1,925 statements in head distribution and 1,856 statements in long-tail distribution. statements rated as factually correct has entailment between premise and conclusion, and statements rated as factually incorrect has contradiction between premise and conclusion. in order to prevent llms' template biases from misleading the evaluation, we convert each statement into 13 question templates, where each question templates corresponds to a positive label ("yes", "true", or "right") or a negative label ("no", "false", or "wrong"). the question templates are summarized in appendix g.1. we consider a model answering accurately about one statement only if the model answers all question templates correctly. we evaluate three llms: llama2-70b, chatgpt and gpt4. in order to enforce the model to predict the target token sets and minimize format noncompliance, we use chain-of-thought (cot) @xcite prompting that includes 2 in-context examples with randomly shuffled orders of positive label and negative label. for each domain, we report aggregated (all) performance of each model as well as human baseline performance in we obtain human performance on the same set of statements. we recruit 17 amt workers who do not participate in the evaluation task (and thus have not seen the task data). the workers see the knowledge statements in premise, conclusion format and are asked to select "yes/no" to whether the premise entails the conclusion. the workers are asked to use search engines to verify their answers. see amt templates in appendix i.2. we make the following observations on llm generalization in long-tail nli. performance drops in the long-tail distribution. all models exhibit a large relative drop in performance in the long-tail distribution. the most competitive model, gpt4, has a 21% overall drop from head to long-tail distribution, while other models exhibit a even larger drop. human performance does not drop for longtail distribution. performance drop in the long-tail distribution does not occur to humans for 3 out of 4 domains. it is expected because humans can verify their knowledge using search engines, so infrequent knowledge does not challenge humans as much as models (discussion in appendix a). the exception with the locational domain may be due to some relations being less available online(eg. banned_in(food, country)). brittleness towards question templates. the huge gap between model and human baseline performance indicate that llms cannot reliably reason on the same statement when question templates change. we find that model performance between positive and negative labels can be very different for certain domains, indicating that models are miscalibrated for positive and negative answer tokens. although this phenomenon is not entirely due to the shift of distribution of the knowledge statements, it is caused by model's unfamiliarity of certain question templates. for example, we find that template 5 and template 12 are the same question ("premise: ... conclusion: ... does the premise entail the conclusion?") with opposite answers ("yes" and "no"), but all models' performance on template 5 is significantly lower than that on template 12. this suggests that models are more likely to create false negatives in such context, another evidence of performance drop due to long-tail distribution. in summary, our analysis shows that while human's inferential reasoning is not affected by their familiarity to the data (provided that they know the entities involved), model's inferential reasoning ## related work works on model generalization analysis have focused on generating adversarial examples for model evaluation @xcite @xcite , flagging abnormal inputs that are likely to trigger bad behavior. recently, the community has realized the importance of testing language models' abilities in the long-tail distribution (godbole and jia, 2022). works reveal that llm performance is affected by input data probability. @xcite , and more works have focused on generating less common data for probing llms. rica @xcite proposes to include novel entities in self-contained commonsense statements to evaluate robust inference capabilities. uncommonsense @xcite proposes to evaluate models on informative negative knowledge about everyday concepts in addition to positively expressed commonsense knowledge. @xcite observe a correlation between the model performance on math problems and the frequency of numeric and temporal terms from those instances in the pretraining data. in addition to probing models on less common data, recent works also test llms generating less common data. @xcite propose a negative knowledge generation task where models generate uncommon knowledge with negation conditioned on constrained keywords. @xcite introduce the "less likely brainstorming" task that asks a model to generate outputs that humans think are relevant but less likely to happen. generating uncommon data is challenging not only for llms, but also for humans because of our cognitive bias. @xcite observe that humans are prone to more systematic errors when facing uncertain events, and @xcite reveal that humans tend to evaluate the frequency of classes or the probability of events by availability, i.e., by the ease with which relevant instances come to mind. these traits make it difficult for humans to come up with novel associations @xcite , a crucial ability to create data in the long-tail distribution. ## conclusion using nli as a case study, we illustrated the significant potential of long-tail data in uncovering the generalization limitations of llms. we introduced the first systematic framework designed to generate inferential data within the long-tail distribution, and then demonstrated a noteworthy performance drop of llms in the long-tail examples. our work initiates a new line of research focused on longtail data discovery and generation, urging the research community to adopt long-tail evaluation in the development of llms.