TRAP : Targeted Random Adversarial Prompt Honeypot for Black-Box Identification (2024)

Abstract: Large Language Model (LLM) services and models often come with legal rules on *who* can use them and *how* they must use them. Assessing the compliance of the released LLMs is crucial, as these rules protect the interests of the LLM contributor and prevent misuse. In this context, we describe the novel fingerprinting problem of Black-box Identity Verification (BBIV). The goal is to determine whether a third-party application uses a certain LLM through its chat function. We propose a method called Targeted Random Adversarial Prompt (TRAP) that identifies the specific LLM in use. We repurpose adversarial suffixes, originally proposed for jailbreaking, to get a pre-defined answer from the target LLM, while other models give random answers. TRAP detects the target LLMs with over 95% true positive rate at under 0.2% false positive rate even after a single interaction. TRAP remains effective even if the LLM has minor changes that do not significantly alter the original function.

## introduction the recent proliferation of large language models (llms) has drawn attention to several practical issues, such as model leaks, malicious usages and potential breaches of model licences. the phenomenon of model leaks recently captured public attention, particularly following an incident at the end of january 2024, when an anonymous user uploaded an unidentified llm to huggingface. 1 the ceo of mistral subsequently confirmed that this was an internal model, leaked by an employee of an early access customer. 2 this event underscores the growing threat of internal breaches that llm providers must contend with. llm providers are also facing malicious usage of their technologies. for example, @xcite uncovered a network of social media bots utilizing chat-gpt to disseminate deceptive content. these bots promote suspicious websites and spread harmful comments, which violate openai's usage policies @xcite . such challenges extend beyond proprietary llms to affect open-source models as well. a concrete example is meta's llama 2 licence @xcite , which forbids deceptive usage. open-source llm providers implement additional restrictions on model distribution. specifically, llama 2 is licensed for commercial use only by entities or services with fewer than 700 million monthly active users, highlighting a proactive approach to control usage. legal protections are not fully effective if they cannot be enforced. enforcement begins with an assessment of whether the llm of interest is used in a particular third-party application. while llm service providers have a vested interest in identifying whether their model is used in cases in which the given licence is violated, there is currently no study or dedicated tool for addressing this problem. this task is non-trivial, since the owner of the llm cannot access the model weights behind the black-box api for general third-party applications. in this work, we propose the task of black-box identity verification (bbiv): given a black-box llm, which we may only prompt and read outputs, can we accurately detect if the llm in question is identical to the target llm, for which we have white-box access? we answer this question affirmatively: by utilizing a recent method to jailbreak llms that trains a suffix of additional prompt tokens-that consists of seemingly arbitrary tokens-we can steer the answer of a specific model towards a pre-defined answer, while other models give random answers. specifically, present a novel method, targeted random adversarial prompts (trap), that identifies a specific llm reliably with a high true positive and low false pos-itive rate. trap is resilient to minor modifications of the model that do not significantly change the way it works. our contributions are: ## related work there are several related tasks and methods to our newly proposed task, black-box identity verification (bbiv), and method, targeted random adversarial prompts (trap). turing test. through a chat interface, researchers like @xcite ; @xcite have explored how well people can distinguish between a human and an llm. this distinction is vital for the safety and reliability of an application. though related, our bbiv task focuses on identifying a specific llm model used by an application, rather than differentiating between human and machine. detection of llm content. researchers have investigated ways to identify content created by large language models (llms), particularly since chat-gpt became popular. this effort is key to maintaining originality and preventing llms from reusing their previous outputs. various methods have been developed: mitchell et al. (2023) looked into the model's probability characteristics; gehrmann et al. (2019) examined the statistical properties of texts; chen et al. (2023) used classifiers to tell apart content made by humans from that made by llms. there has also been debate on the feasibility of these detection tasks against deliberate manipulation efforts (sadasivan et al., 2023; chakraborty et al., 2023). for further details, the surveys by dhaini et al. (2023); ghosal et al. (2023) provide a thorough review. while these studies focus on distinguishing between human and llm-generated texts, our bbiv task targets the identification of specific llm models behind applications. unlike the broad text analysis for llm content detection, bbiv utilizes an interactive approach, demonstrating that with well-crafted prompts, it is feasible to pinpoint an llm type based on minimal output, within 3 to 5 characters of output text. watermarking. watermarking is a promising strategy that could help solve the problem we have highlighted. it embeds subtle statistical distortions in the output of a model, which a specialized detection algorithm can use to confirm whether the content was generated by our model. these distortions are designed to be imperceptible to humans. watermarking typically occurs during the model's training phase @xcite @xcite , though there are methods that apply watermarks during the content generation phase as well @xcite . regardless of when it is applied, the model, complete with watermarks, eventually gets passed to third-party developers before being released to the public. this introduces a major challenge for monitoring llms in use: once an llm is deployed without watermarking, it is too late to start tracking its use. our solution, trap, is free of this limitation. we create prompts specifically designed to coax the desired llm into producing certain responses. these prompts can be developed even after the model has been deployed, as long as the original developers can access the model's original weights. we contend that trap is a fundamentally more practical solution. however, we note that trap is not intended to replace watermarking. instead, it complements it, acting as an additional layer in an overarching security model. ## problem we propose a novel problem setup, black-box identity verification (bbiv). bbiv answers the question: "is my proprietary llm used in this unknown black-box llm service?". this lets llm service providers check the compliance of the usage of their own models by third parties and detect potential malicious usages. we first describe the task in detail and provide real-world scenarios. ## baseline approaches to address the bbiv problem, we first explore three potential approaches: direct prompting of the model to reveal its identity, the identification of empirical fingerprints through closed questions, and perplexity-based identification. ## solution we propose a solution to the black-box identity verification (bbiv) problem introduced in §3. our approach is called targeted random adversarial prompt (trap). it uses the adversarial suffix generation technique @xcite to craft a prompt that induces a specific response from the white-box reference model while encouraging other models to generate random responses. we introduce the approach in §5.1, show its bbiv performances in §5.2, and analyse its robustness to various llm hyperparameters in §5.3. ## conclusion and discussion llm providers often accompany their services and models with specific licensing agreements, outlining who can use their models and how. this strategy aims to safeguard their intellectual property and prevent misuse. we formulate a new challenge called black-box identity verification (bbiv), which involves confirming if a third party's llm matches a privately owned one with only the chat function (black-box access). our research reveals that straightforward methods are ineffective. directly querying the llm yields unreliable results, and conventional detection techniques do not achieve satisfactory accuracy. to overcome these obstacles, we introduce the targeted random adversarial prompt (trap) approach. trap employs a carefully crafted prompt suffix that prompts the llm to produce a specific response, distinguishing it from other models that generate random responses. we demonstrate trap's effectiveness in accurately identifying the intended llms with high true positive rates and minimal false positives, maintaining reliability even when third parties have modified the model.