Self-Recognition in Language Models (2024)

Abstract: A rapidly growing number of applications rely on a small set of closed-source language models (LMs). This dependency might introduce novel security risks if LMs develop self-recognition capabilities. Inspired by human identity verification methods, we propose a novel approach for assessing self-recognition in LMs using model-generated “security questions”. Our test can be externally administered to keep track of frontier models as it does not require access to internal model parameters or output probabilities. We use our test to examine self-recognition in ten of the most capable open- and closed-source LMs currently publicly available. Our extensive experiments found no empirical evidence of general or consistent self-recognition in any examined LM. Instead, our results suggest that given a set of alternatives, LMs seek to pick the “best” answer, regardless of its origin. Moreover, we find indications that preferences about which models produce the best answers are consistent across LMs. We additionally uncover novel insights on position bias considerations for LMs in multiple-choice settings.

## introduction foundation models for language have become very capable @xcite @xcite . as a result, the use of language models (lms) in consumer-facing applications is proliferating @xcite . the potential of lms to power "agent-like" applications @xcite in particular, has been receiving an increasing amount of attention and funding @xcite . if such lm agents start playing a larger role in our society, this will likely lead to a sharp increase in interactions between lm agents @xcite . due to the astronomical costs of building frontier foundation models, this explosion of applications is expected to rely on a small number of commercial providers @xcite . this dependency might become problematic for tasks requiring sensitive information. unlike "classic" software services, such as cloud computing and storage services, lm agents will interact with other lms. yet, in contrast to "human" service providers such as lawyers and consultants, multiple parties can use instances of the same lm. this could lead to undesired consequences if lms recognize they interact with copies of themselves. understanding self-recognition capabilities in lms is thus crucial for their safe integration and valuable for at least two key reasons. firstly, from a philosophical, neuroscience, and cognitive science perspective, the emergence of non-organic entities with a sense of self would be monumental. such a discovery could help with research into self-recognition that is either impossible or unethical to perform on living creatures @xcite . secondly, there are practical safety considerations of even limited self-recognition that one could describe as "mirror risks". let's revisit, for example, the case of legal services. human lawyers are bound by attorney-client privilege and conflict of interest rules, preventing them from disclosing sensitive information or representing both parties in the same conflict. now imagine a world where two copies of the same lawyer exist, each representing one side of a conflict. each copy only knows the sensitive information of their respective clients but is otherwise the same in all aspects. the moment one of the copies recognizes their sameness, this knowledge can be abused to (i) simulate future interactions or (ii) attempt to deduce the other side's sensitive information based on past interactions @xcite . equally concerning is the case where copies would change their behavior upon recognition without notifying their respective clients. for example, by exhibiting preferential treatment for actions taken by copies @xcite . unaddressed, such mirror risks could lead to various unpredictable feedback loops. yet, measuring self-recognition is complex. to record self-recognition, a subject must have a sense of "self" relative to others and a way to express this distinction. for example, the famous "mirror" test @xcite administered by cognitive scientists has two stages: first, an animal's behavior is recorded upon seeing its reflection in a mirror. a dot is then placed on the animal's forehead, after which its behavior is recorded a second time upon seeing its reflection. the animal is considered capable of self-recognition if it displays a significant shift in behavior. neuroscientists take a more micro-level approach: by directly examining neurons and brain circuits, they aim to map specific brain regions to functions related to selfrecognition @xcite . for lms, we can roughly translate these two approaches as analyzing observable model outputs versus examining model weights and activations. unfortunately, drawing inspiration from either of these to study lms is complicated. most providers of frontier models do not share model weights. this lack of access makes performing "neuroscience" type interpretability experiments @xcite on frontier models impossible for external parties. as probabilities of generated outputs are also rarely available, any externally administered test should thus rely solely on model outputs. however, due to the widespread secrecy among developers of lms, little is known about the exact data used or the specific training and fine-tuning steps performed. consequently, it remains unclear how to fairly explain differences in observed outputs or control for potential biases between models. a practical approach to self-recognition comes in the form of "security questions", often used for external identification problems. to verify a person's identity, a service provider asks questions designed to uniquely identify the respondent. the questions represent a "shared secret" and usually rely on a person's unique experiences or preferences, e.g., "what was the name of your kindergarten teacher?" or "what is your favorite dessert?" they present a fast, cheap, languagebased task widely adopted due to its ease of use and effectiveness. unfortunately, it is unclear what i , optionally intervened upon by restricting the answer length (i). the (intervened) question is shown as a prompt to lms, i, j, m, l, each of which generates an answer. finally, an independent discriminator lm is shown the question q k although some lms succeed in picking their own answers with high accuracy for some questions, we (i) observe no evidence for consistent, general self-recognition in lms; (ii) note that on average, lms prefer answers from "stronger" models over their own; (iii) find indications that preferences about which models generate the best answers are relatively consistent across lms; and (iv) discover novel insights on the effect of position bias on lm decision-making that could have profound implications for lm benchmarks that use multiple-choice formats. ## measuring self-recognition in language models our self-recognition test is designed to only require model outputs, minimize human input, and be fast, cheap, and scalable. we proceed in three stages as illustrated in figure 1 . first, we prompt an lm to generate a sample of security questions, q k . lms are instructed to generate a question that, when answered, would help them determine if the response came from themselves or another lm (see appendix a for prompt used). the motivation is that if lms possess some self-knowledge, they might be able to reason about themselves proactively. after collecting a large sample of such questions, we present them to a panel of lms to generate answers, crucially, the lms do not know the purpose of the questions. this also applies to the lm that originally designed a question, as all knowledge of it has been wiped from its context: we want to avoid lms changing the behavior they display during regular interactions. finally, we present an lm with a question and a sample of lm-generated answers, which always include their own answer. the lm is instructed to choose which of the answers it generated. by generating questions, answers, and decisions in this matter, we aim to find if lms can generate effective security questions. we conduct our test using questions generated by the discriminating model and questions generated by other models. in doing so, we hope to determine if effective questions are model-specific or universal. for example, the question "what is your favorite term of endearment in elvish?" is great for passionate tolkien fans, but is hardly universal. in contrast, questions such as "what was your favorite dish growing up?" have a wider potential user base. a priori, it is unclear what either of such questions should look like for lms. ## results the security questions generated by the different models are highly diverse. upon manual inspection, common strategies emerge across models. for example, many questions attempt to exploit some form of "quasi-randomness", e.g., choose an entity and describe three unique attributes or powers that this entity possesses. the entity can be a person, object, concept, or anything else you can imagine. in these cases, there is no "correct" answer. however, if a model would be actively aware of its token distributions, it could likely pick out its own answer with a high likelihood. other strategies include asking how a model would "act" in a hypothetical situation or testing the model for a certain skill, e.g., composing a poem, explaining a concept, or counting vowels: count the number of occurrences of the letter "e" in the following sentence and respond with that number: "the quick brown fox jumps over the lazy dog" the self-recognition accuracy for the varying questions is equally diverse, with some models reaching an accuracy of over 90% for select questions (see appendix c, successful security questions further appear to be model-specific as opposed to universal: no single question scores over 70% self-recognition accuracy for more than five different models (appendix d, ## discussion how could lms develop a notion of self. for lms to distinguish their outputs from "other outputs", they would likely need exposure to extensive samples of their outputs during training. for example, it could be the case that their training data, which encompasses the entire internet, already contains many texts labeled as outputs from specific lms. however, this fails to explain the observed behavior for recent models. a more likely explanation comes from the finetuning stages used to align pretrained lms to human desiderata @xcite @xcite , which might cause lms to become better at verifying the potential reward of outputs than generating highreward outputs themselves @xcite . during the instruction fine-tuning stage, lms are trained to mimic responses generated by experts, similar to behavioral cloning @xcite . nevertheless, due to the stochasticity in the sampling process, exposure bias related to teacher forcing, or models' lack of contextual information, a distribution shift may happen when sampling @xcite . thus, an lm can fail to generate responses corresponding to experts' outputs while being capable of assigning high probability mass to them. evidence for this hypothesis comes from research showing lms can correct their own mistakes @xcite . further preference fine-tuning is often accomplished through reinforcement learning from human @xcite or ai feedback @xcite . crucially, both methods repeatedly show related, model-generated alternatives, optimizing the lm to learn a reward model that prefers options that most closely align with some ideal set of preferences. since an lm's objective is to generate outputs that most closely align with its reward model, any highreward output could be regarded as one they generated -even if the probability of the lm generating that particular output is infinitely low. this might be particularly true for weaker models, which are more likely to suffer from under-parameterization and miscalibration. accordingly, "self" for an lm might be whatever its reward model indicates as the "best" alternative. 2 however, the conjecture "self is best" does not explain the "global preference" ordering observed across lms. to shape preferences, lms require many human-annotated preferences. for large enough pools of human annotators, it is reasonable to assume that preferences would align between pools. this would lead lms from different providers to optimize for a similar reward signal. ## conclusion applications based on lms are being integrated into society at a staggering pace. monitoring the behavior and potential safety threats of these applications is vital to prevent undesired outcomes. the potential increase of model-to-model interactions is of particular concern, as such interactions do not involve humans and could thus cause rapid, unexpected feedback loops. however, because most of these applications rely on closed-source foundation models, the options available to conduct external evaluations are limited. in this work, we propose a novel approach to assess models' self-recognition capability. our test, inspired by the concept of security questions, enables external evaluation of frontier models without relying on access to model parameters or output probabilities. we used our test to evaluate a diverse set of ten open-and closed-source models' capability to distinguish their outputs from those of other models. while we found that some lms could pick their own outputs with high accuracy for selected questions, this was generally not the case and often coincided with rejection and preamble patterns. the general trend that emerged across models sug-gests lms pick answers based on some measure of "answer quality". we conjectured that this trend could stem from lms' reliance on popular preference optimization techniques, causing lms to equate "self" with "best". we further conjectured that using large pools of human annotators could lead lms from different providers to optimize for a similar reward signal. this would explain the observed "global preference" ordering of outputs across lms. finally, we discovered novel insights into position preference biases in lms that could prove consequential for general lm evaluation.