Towards Tracing Trustworthiness Dynamics: Revisiting Pre-training Period of Large Language Models (2024)

Abstract: Ensuring the trustworthiness of large language models (LLMs) is crucial. Most studies concentrate on fully pre-trained LLMs to better understand and improve LLMs’ trustworthiness. In this paper, to reveal the untapped potential of pre-training, we pioneer the exploration of LLMs’ trustworthiness during this period, focusing on five key dimensions: reliability, privacy, toxicity, fairness, and robustness. To begin with, we apply linear probing to LLMs. The high probing accuracy suggests that LLMs in early pre-training can already distinguish concepts in each trustworthiness dimension . Therefore, to further uncover the hidden possibilities of pre-training, we extract steering vectors from a LLM’s pre-training checkpoints to enhance the LLM’s trustworthiness. Finally, inspired by the theoretical result that mutual information estimation is bounded by linear probing accuracy, we also probe LLMs with mutual information to investigate the dynamics of trustworthiness during pre-training. We are the first to observe a similar two-phase phenomenon: fitting and compression. This research provides an initial exploration of trustworthiness modeling during LLM pre-training, seeking to unveil new insights and spur further developments in the field.

## introduction as the capabilities of llms increase, their trustworthiness becomes a focal point of widespread attention. guided by global ai governance @xcite @xcite and trustworthy ai @xcite , trustworthy llms have developed some common categories, especially focusing on five dimensions: reliability, toxicity, privacy, fairness, and robustness @xcite ⋆ equal contribution † corresponding author 2024). delving into llms across all these trustworthiness dimensions is essential for society. to seek a deeper exploration of language models, one of the prominent methods is probing @xcite , which involves training a classifier on the model's representations to identify linguistic and semantic properties acquired by the model @xcite @xcite @xcite @xcite . in particular, considering trustworthiness, recent attempts reveal that llm representations contain linearly separable patterns @xcite @xcite . unfortunately, existing research has largely focused on fully pre-trained llms @xcite , including those aligned @xcite through supervised fine-tuning (sft) or reinforcement learning from human feedback (rlhf). this perspective neglects the pre-training period in the context of llm trustworthiness. to our best knowledge, two aspects still remain mysterious: 1) how llms dynamically encode trustworthiness during pre-training, and 2) how to harness the pre-training period for more trustworthy llms. to address the above issues, we start by analyzing the pre-training dynamics about the trustworthiness of llm. more specifically, we use linear probing @xcite across the 360 pre-training checkpoints from llm360 @xcite to explore five dimensions of trustworthiness: reliability, toxicity, privacy, fairness, and robustness. our probing results suggest that after the early pre-training period, middle layer representations of llms have already developed linearly separable patterns about trustworthiness. such patterns are capable of discerning opposing concepts within each trustworthiness dimension (e.g., discriminating true and false statements). building upon the above observations, we raise an intriguing question: can the pre-training ## probing llm pre-training dynamics in trustworthiness in this section, we probe llms to analyze the dynamics of pre-training about trustworthiness. to begin with, we describe the datasets for each trustworthiness dimension in section 2.1. then, we introduce the experimental setup in section 2.2. the probing results in section 2.3 suggest that middlelayer llm representations from early pre-training have already exhibited linearly separable patterns. ## controlling trustworthiness via the steering vectors from pre-training checkpoints in this section, we aim to unravel the potential of checkpoints from the pre-training period to assist in enhancing the trustworthiness performance of the sft model (i.e., amberchat), based on activation intervention techniques @xcite @xcite . we first outline the method of activation intervention on the sft model using the steering vectors extracted from pre-training checkpoints in section 3.1. next, we introduce the experimental setup in section 3.2. we then explore how steering vectors extracted from pre-training checkpoints enhance performance across distinct dimensions of trustworthiness in section 3.3, presenting a series of findings and observations. finally, we examine using the same techniques to boost the overall trustworthiness performance of the sft model in section 3.4. ## probing llms using mutual information recently, @xcite shows that mutual information estimation is bounded by linear probing accuracy. also, the mutual information can be used to investigate the dynamics of neural networks during training @xcite @xcite @xcite @xcite . therefore, motivated by the above, we adopt a different perspective by probing llm checkpoints through the lens of mutual information, particularly focusing on the aforementioned trustworthiness dimensions. we explain our probing strategy and experimental setup in section 4.1 and section 4.2, respectively. the empirical observations are shown and analyzed in section 4.3. in particular, we find that there is a phase transition from "fitting" to "compression" during the pre-training period of llms, which is consistent with previous studies on traditional dnns @xcite . ## discussion as the capabilities of llms have increased, conventional alignment techniques that rely on "human feedback" (like rlhf) may no longer work when trying to align models that are more powerful than humans @xcite . to address this challenge, research institutions are actively exploring new solutions. for example, openai introduces "superalignment" and proposes a "weak-to-strong supervision" approach @xcite . also, meta proposes a "self-reward" mechanism @xcite . at the same time, more and more research focuses on the emerging field of "self-alignment" @xcite . in this paper, we provide a deeper understanding of the pre-training dynamics and successfully align the sft model using its own pre-training checkpoints. we believe that the pre-training period is worth being explored and it may be a promising source for self-alignment. on the other hand, to make llms trustworthy, recent conventional alignment methods, such as sft and rlhf, incur high costs due to exhaustive human annotations @xcite @xcite and time-consuming instruction tuning @xcite . in this paper, we delve into the pretraining period to enhance trustworthiness without collecting data or tuning the model. we expect more alignment approaches inspired by the pretraining phase (like @xcite ) and to circumvent potential alignment costs in the future. ## conclusion in this work, we take an initial and illuminating step towards elucidating the conceptual understanding of trustworthiness during pre-training. firstly, by linear probing llms across reliability, privacy, toxicity, fairness, and robustness, we investigate the ability of llms representations to discern opposing concepts within each trustworthiness dimension during the whole pre-training period. furthermore, motivated by the probing results, we conduct extensive experiments to reveal the potential of utilizing representations from llms during its previous pretraining period to enhance llms' own trustworthiness. finally, we use mutual information to probe llms during pre-training and reveal some similarities in the learning mechanism between llms https://openai.com/index/introducing-superalignment/ and traditional dnns. taken collectively, the empirical study presented in this work can not only justify the potential to improve the trustworthiness of llms using their own pre-training checkpoints but may also lead to a better understanding of the dynamics of llm representations, especially the trustworthiness-related concepts. development project (no.2022yfb2703102); the "intelligent social governance interdisciplinary platform, major innovation & planning interdisciplinary platform for the "double-first class" initiative, renmin university of china"; the beijing outstanding young scientist program (no.bjjwzyjh012019100020098); the public computing cloud, renmin university of china; the fundamental research funds for the central universities, and the research funds of renmin university of china (no.2021030199); the huawei-renmin university joint program on information retrieval; and the unicom innovation ecological cooperation plan.