Impact of Co-occurrence on Factual Knowledge of Large Language Models (2023)

Abstract: Large language models (LLMs) often make factually incorrect responses despite their success in various applications. In this paper, we hypothesize that relying heavily on simple co-occurrence statistics of the pre-training corpora is one of the main factors that cause factual errors. Our results reveal that LLMs are vulnerable to the co-occurrence bias, defined as preferring frequently co-occurred words over the correct answer. Consequently, LLMs struggle to recall facts whose subject and object rarely co-occur in the pre-training dataset although they are seen during finetuning. We show that co-occurrence bias remains despite scaling up model sizes or finetuning. Therefore, we suggest finetuning on a debiased dataset to mitigate the bias by filtering out biased samples whose subject-object co-occurrence count is high. Although debiased finetuning allows LLMs to memorize rare facts in the training set, it is not effective in recalling rare facts unseen during finetuning. Further research in mitigation will help build reliable language models by preventing potential errors. The code is available at https://github.com/CheongWoong/impact_of_cooccurrence.

## introduction natural language processing has seen significant progress in recent years with the advent of large language models (llms) @xcite @xcite . factual knowledge probing benchmarks like lama have demonstrated that llms have a high capacity to recall factual knowledge @xcite @xcite @xcite . however, factual knowledge stored in llms may not always be correct @xcite . understanding the reasons behind such inaccuracies is critical for developing more accurate and reliable language models. recent studies point out that llms often learn short- the capital of canada is [mask] language model pre-training data toronto ottawa subject object count canada toronto 246 canada ottawa 19 … … … figure 1 : this figure shows an overall framework of our correlation analysis between co-occurrence counts and factual knowledge of llms. we assume that if the target model heavily relies on subject-object co-occurrence, it is more likely to recall the most co-occurring word without accurate semantic understanding. for instance, in this hypothetical example, the model fails to answer the question about the capital of canada by generating the most frequently co-occurring word 'toronto', while the correct answer is 'ottawa'. this indicates that relying heavily on co-occurrence statistics may have potential errors. in this work, we investigate the effects of cooccurrence statistics of the pre-training data on factual knowledge in llms. first, we adopt the lama dataset @xcite to probe factual knowledge, represented as a subject-relationobject triple. then, we analyze the correlation between co-occurrence statistics and performance on factual knowledge probing. specifically, we count co-occurrences of word pairs in the pre-training corpora. we focus on subject-object co-occurrence, motivated by the concept of distant supervision, which shows that a sentence often contains the triple if it contains a subject and an object of a triple @xcite . figure 1 illustrates an overall framework of our correlation analysis between cooccurrence counts and factual knowledge of llms. we hypothesize that the target model would generate the most frequently co-occurring word if it heavily relies on co-occurrence. in this simulated example, given the fact 'canada'-'capital'-'ottawa', the target model generates the most frequently cooccurring word 'toronto', which is not the correct answer. we test our hypothesis with gpt-neo @xcite and gpt-j @xcite , which are open-source versions of gpt-3 @xcite . we compute co-occurrence statistics of the pile dataset @xcite , on which the target models are pre-trained. we show that the factual probing accuracy of llms highly correlates with subject-object co-occurrence, leading to failures in recalling rare facts. although scaling up model sizes or finetuning boosts the overall performance on factual knowledge probing, they do not resolve co-occurrence bias, in which frequently co-occurred words are preferred over the correct answer. besides, we find that a significant portion of facts in the lama dataset can be recalled by simply generating the object with the highest co-occurrence count. although cooccurrence is necessary to recall factual knowledge, it is not sufficient. therefore, relying heavily on co-occurrence is inappropriate for understanding the accurate meaning behind words. relying heavily on the co-occurrence statistics may lead to hallucinations @xcite @xcite if the co-occurrence statistics reflect factually incorrect information. therefore, we suggest finetuning llms on the debiased lama, constructed by filtering out biased samples whose subject-object co-occurrence count is high. although finetuning on the debiased dataset allows llms to learn rare facts that appear in the training set, it is not generalizable to test cases. in summary, we show that factual knowledge probing accuracy correlates with subject-object co-occurrence. in addition, we present novel ev-idence and insights by providing a more detailed picture. specifically, we demonstrate that llms prefer frequently co-occurring words, which often override the correct answer, especially when the correct answer rarely co-occurs with the subject. while existing studies only show that the performance of llms correlates with co-occurrence, our results provide evidence and reasons for that. we hope our results spur future work on mitigating co-occurrence bias to build accurate and reliable language models. ## factual knowledge probing this section describes the overall framework to test factual knowledge of llms. with this framework, we aim to investigate the effects of model sizes, finetuning, and co-occurrence statistics. ## factual knowledge probing with co-occurrence statistics this section describes our framework to analyze the impact of pre-training co-occurrence statistics on factual knowledge of llms. we first test how much factual knowledge can be recalled with term frequency statistics, including co-occurrence. then, we analyze the correlation between cooccurrence statistics and factual predictions. ## mitigation debiasing with undersampling considering facts whose subject-object co-occurrence count is high as biased samples, we suggest finetuning lms on a debiased dataset, constructed by filtering out biased samples from the training set. given a dataset d with samples x i and corresponding bias we report the micro-average hits@1 of gpt-j 6b on the original training set in the remove stopwords setting, comparing debiased finetuning and the random filtering baseline. the results show that debiased finetuning helps to learn rare facts but hampers learning frequent facts. (b) effects of filtering ratios: we observe that higher filtering ratios cause performance degradation on frequent facts but marginal improvement on rare facts. therefore, a filtering ratio needs to be properly tuned to maximize the performance gains on rare facts while keeping the performance on frequent facts. we also analyze the effects of debiased finetuning on the test set. the results show that the effect of debiased finetuning is marginal at test time. scores score bias (x i ), and a filtering ratio p, we discard p% of the total samples with the highest bias scores. we compute score bias (x i ) as the conditional probability p pretrain (obj i |subj i ). since the number of samples is reduced, we consider a random filtering baseline that randomly filters out p% of the total samples for a fair comparison. note that we apply the filtering algorithms for each relation separately to prevent discarding nearly all samples of a highly biased relation. we test three different filtering ratios: 0.1, 0.3, and 0.5. the results of gpt-j 6b on the original training set are shown in figure 8 . figure 8a compares the debiased model with the random filtering baseline with a filtering ratio 0.5. the debiased model surpasses the baseline on rare facts with sacrifices on frequent ones. figure 8b compares the effects of different filtering ratios on debiased finetuning. we observe that the performance on frequent facts significantly degrades while improvements on rare ones are marginal as more samples are filtered out. furthermore, we investigate the effects of debiased finetuning on the test set in figure 9 . we observe that the performance of the debiased model and the baseline are similar regardless of co-occurrence counts, implying that the effect of debiased finetuning is not generalizable. since it is non-trivial to directly fix the cause of co-occurrence bias, designing a more sophisticated debiasing algorithm or using other approaches may be beneficial to complement the proposed debiased finetuning. we leave further investigation in this direction as future work. ## conclusion in this work, we investigate the impact of cooccurrence statistics of the pre-training corpora on factual knowledge of llms. we reveal the co-occurrence bias of llms, in which they prefer words that frequently co-occur with the subject entity over the correct answer. as a result, llms struggle to recall facts whose subject and object rarely co-occur in the pre-training corpora although they are seen during finetuning. although scaling up model sizes or finetuning substantially improves the overall performance, the co-occurrence bias remains. therefore, we suggest further investigation on mitigating co-occurrence bias to ensure the reliability of language models.