Formality is Favored: Unraveling the Learning Preferences of Large Language Models on Data with Conflicting Knowledge (2024)

Abstract: Having been trained on massive pretraining data, large language models have shown excellent performance on many knowledge-intensive tasks. However, pretraining data tends to contain misleading and even conflicting information, and it is intriguing to understand how LLMs handle these noisy data during training. In this study, we systematically analyze LLMs’ learning preferences for data with conflicting knowledge. We find that pretrained LLMs establish learning preferences similar to humans, i.e., preferences towards formal texts and texts with fewer spelling errors, resulting in faster learning and more favorable treatment of knowledge in data with such features when facing conflicts. This finding is generalizable across models and languages and is more evident in larger models. An in-depth analysis reveals that LLMs tend to trust data with features that signify consistency with the majority of data, and it is possible to instill new preferences and erase old ones by manipulating the degree of consistency with the majority data.

## introduction large language models (llms) such as llama @xcite , chatgpt and gpt4 @xcite have revolutionized the landscape of natural language process research, and are shown to possess massive world knowledge @xcite @xcite , even surpassing human-level performance in various knowledge-intensive benchmarks @xcite @xcite . nearly all knowledge of llms comes from the pretraining corpus, a large amount of which are web-crawled. although rigorously cleaned, they still inevitably contain misleading and even conflicting information. it is intriguing how llms deals with these noisy data. when encountering conflicts of knowledge in a text, human beings can leverage additional perspectives, such as information sources or consistency with more information, to aid in their judgments. as llms have accumulated a large amount of common sense knowledge in their parameters, it is interesting to investigate whether llms have developed similar strategies when faced with conflicting knowledge from different texts. in this paper, we present a systematic study on the learning preferences of llms, i.e., the strategies they use to choose between texts with specific features when facing conflicting knowledge in the training corpora. we first construct our own biographical pseudo-data with conflicting knowledge. then, we fine-tune llms on data with specified features, ensuring that data with different features contain conflicting knowledge. the preference for different data features in model fine-tuning can be identified by calculating the degree of preference of the llms after fine-tuning. empirically, we find that pretrained llms exhibit notable learning preferences towards specific textual features. these preferences are reflected in two ways: (1) at training time, llms learn faster on data with more preferred features; (2) at test time, llms assign larger probability to knowledge in data with more preferred features. concretely, llms prefer formal styles, such as scientific reports and newspaper styles, rather than relatively casual expressions, such as social media and novel styles. this preference for stylistic features arises as the model scale increases and is observed across different llms and in different languages. we also observed that spelling errors in the training data lead to negative preferences in the model, a phenomenon that is prevalent across multiple models in multiple languages. observing that preferred ## related work understanding the mechanism of knowledge learning for llms. there are a handful of works that aim to understand the mechanism of knowledge learning for llms. many works attempt to understand how knowledge is stored and retrieved in the llms' parameters. jawahar et al. (2019 2022 ) further search for neurons that are causally related to specific knowledge using the integrated gradient method and causal tracing @xcite . compared to these works, our paper mainly focuses on how the presentation of knowledge affects the learning process. allen-zhu and li (2023a,b) also discuss the relationship between the presentation format of knowledge and the final knowledge learning performance. they find that adopting knowledge augmentation, e.g., paraphrasing, during the pretraining stage substantially improves the downstream question answering performance on knowledge-related tasks. we follow this strategy in our paper and investigate how high-level features, e.g., style, spelling correctness, and consistency with other data, affect the learning process. ## conclusion in this paper, we investigate the learning preferences of large language models. thorough extensive experiments on synthetic biographies data, we reveal that existing pretrained large language models have established preferences as human beings do, e.g. preferring formal texts and texts with less spelling errors. we also provide an initial attempt to explain how such preferences is developed, i.e. llms can effectively identify features that signify the degree of consistency between current text and the remaining data, and use such features to determine whether the current text is worth learning. we hope our work could provide a new perspective to study the knowledge learning mechanism of llms.