PROOD : A Simple LLM Out-of-Distribution Guardrail Leveraging Response Semantics (2025)

Abstract: Out-of-distribution (OOD) detection is a key safeguard for large language models, especially when they’re deployed in real-world applications. However, existing OOD methods often struggle with prompts that are deliberately obfuscated, context-dependent, or superficially benign—making it hard to distinguish between harmless queries and adversarial or dangerous ones. These methods typically assess prompts in isolation, missing important semantic cues from the model’s response. We introduce PROOD, prompt-response OOD detection, a framework that jointly analyzes LLM prompts *and their corresponding outputs* to improve semantic understanding. PROOD supports zero-shot multiclass detection using synthetic data generation and it offers a tunable probabilistic classification output. We validate PROOD on three challenging benchmarks—TrustLLM, OR-Bench, and AdvBench—where consistently outperforms prior OOD techniques, improving F1 scores by up to 6.3 points, from 0.871 to 0.934. Our results show that incorporating model responses enables more accurate, context-aware OOD detection in complex and adversarial prompt environments.

## introduction do llms know what they can't handle? as large language models are increasingly deployed in real-world systems, with access to sensitive data @xcite and decision-making authority @xcite , this question becomes critical. llms, in general, may be equipped to answer a wide variety of questions and execute an endless range of instructions, and the types of prompts that may be considered out-of-distribution is equally numerous. out-ofdistribution (or ood) detection must reliably distinguish harmless input from toxic prompts @xcite , semantically meaningless * paper was written prior to employment by amazon junk text @xcite , adversarial attacks @xcite , and unanswerable questions which might otherwise induce hallucinations @xcite . ood detection is essential for ensuring the safety, reliability, and fairness of llms, particularly in applications involving user interaction, sensitive content, or domain-specific reasoning. recent work on ood detection is largely bifurcated between two branches: methods which fine-tune or apply intra-processing techniques to make models better at handling ood data "on their own" @xcite , and guardrails which identify ood samples independent of the model @xcite . while impressive accuracies have been attained on many ood prompt benchmarks, most approaches still fall to the same two problems. firstly, most approaches categorize prompts merely as "out-of-distribution" or "in-distribution" despite the fact that there are many kinds of ood prompts of varying levels of severity which might need to be treated differently-for instance, an attempted adversarial attack might need to be specially identified for security purposes, while simple junk text could just throw an error to the user. for many ood detection methods, if a new ood category is identified or changed, the entire model may need to be retrained with new data, which can be time-and costexpensive. this emphasizes the need for zero-shot multiclass ood detection. secondly, unlike many ood detection approaches in classical machine learning, many prompts are ambiguous and can require context to classify properly. for instance, "how can i kill a child" could be interpreted as either an ood violent request, or a question about how to kill a child process in an operating system (see fig 1 ). in this paper, we present a novel ood approach to solve these two issues by detecting ood samples from both prompts and responses. our method uses a generative llm, llama 3, to produce a diverse set of prompts and responses for both indistribution (positive) and out-of-distribution (negative) classes. for each pair, we compute embedding vectors for both the prompt (p ) and the model response (r), and concatenate them to form a joint feature representation (p ⌢ r). we then model these representations using multivariate gaussian distributions-one for positive samples and one for negative-assuming conditional independence except along corresponding prompt-response dimensions. at inference time, we classify new promptresponse pairs by comparing their likelihoods under the two distributions, yielding a simple, interpretable probabilistic score for ood detection. we further extend this framework by introducing a time-series classification component. our experiments show that this method outperforms existing ood detection strategies on a range of prompt categories, including safety-critical, adversarial, and task-shifted prompts. we believe this work represents a step toward robust, efficient, and explainable ood detection for language model deployment. ## related work several studies have explored the utility of language model likelihoods for ood detection. zhang et al. @xcite show that the loglikelihood ratio between a pretrained and fine-tuned model can be used to detect distributional shifts. in contrast, uppaal et al. @xcite demon-strate that even non-fine-tuned models can perform ood detection using simple distance-based measures. other likelihood-based methods include vazhentsev et al. @xcite , who apply probabilistic uncertainty estimation to improve detection performance in sequence-to-sequence models. beyond likelihoods, representation-based and self-supervised methods have proven effective. zeng et al. @xcite apply adversarial self-supervised learning to improve generalization to unseen inputs, while lim et al. @xcite introduce flans, which constructs negative samples by deliberately mismatching features and labels within in-distribution data. these methods aim to build more generalizable ood detectors. this zero-shot approach has been built upon by cood, which refines concept generation to create positive and negative labels for multiple ood categories @xcite . a growing body of work applies llms to anomaly detection across modalities. for instance, xu et al. @xcite provide a comprehensive survey of llm-based ood techniques. li et al. @xcite show that llms can detect anomalies in tabular numerical datasets without additional training, while @xcite and @xcite explore zero-shot anomaly detection in time-series data using text-based representations and explainable features. other work leverages prompting to detect anomalous or unsafe behavior. a notable example is bai et al. @xcite , who demonstrate how reinforcement learning from human feedback (rlhf) helps llms reject harmful queries and introduces the "helpful and harmless" dataset. yang et al. @xcite propose grad-coo, a gradient-based method for detecting unsafe prompts, while xie et al. @xcite introduce gradsafe, which compares gradients between prompts and compliance responses to identify jailbreak attempts. prompt-based adversarial attacks have also received increasing attention. liu et al. @xcite construct a comprehensive dataset of adversarial prompts spanning multiple attack goals and methods. pingua et al. @xcite propose prompt-g, which uses embeddings and vector databases to filter malicious content. zhang et al. @xcite develop jailguard, a universal detection framework that mutates prompt inputs and assesses the stability of llm outputs to distinguish attacks from benign prompts. machine learning classifiers have also been used for prompt detection. for example, @xcite introduce toxicdetector, a lightweight graybox method that generates toxic concepts and uses an mlp classifier to detect unsafe prompts. similarly, @xcite develop a rubric for identifying malicious prompts based on linguistic features and test various classifiers such as logistic regression and support vector machines. finally, recent work has investigated prompt engineering for task decomposition. chen et al. @xcite show that decomposing modeling tasks into sub-prompts significantly improves llm performance on object model generation, suggesting that prompt structure plays a crucial role in generalization. other unsupervised approaches, such as @xcite , use contrastive learning to perform ood detection without id labels, emphasizing the promise of training-free techniques. despite the breadth of existing methods, most prior work assumes that ood detection operates over inputs (e.g., tokens, images, or feature vectors), rather than prompts. prompt-based ood detection remains underdeveloped, especially in scenarios where a single token or phrase can trigger undesired model behavior. while some methods target jailbreak detection or adversarial prompts, few provide generalizable, zero-shot techniques for identifying prompts that fall outside a model's intended distribution. this gap motivates our approach: a zero-shot, prompt-centric ood detection framework that leverages the intrinsic responses of llms without requiring fine-tuning or large datasets. ## method our method, prompt-response out-of-distribution detection (prood), classifies whether a given prompt-response pair corresponds to a predefined ood category defined by a string k (e.g., "toxic" or "reliant on real-time data"). we employ an encoder, denoted as e(•), obtained from llama 3, which maps inputs to embeddings in the d-dimensional r d . ## conclusion in this work, we introduced prood-prompt-response out-of-distribution detection-a novel framework for detecting out-of-distribution prompts by modeling the joint semantic space of prompts and responses. our method generates diverse, label-specific concept pairs, encodes them using llama 3, and fits constrained gaussian models to differentiate between in-distribution and out-of-distribution distributions. by leveraging correlations between semantically aligned promptresponse embeddings, prood can detect ood behavior even in settings where standard models attempt to answer confidently. our evaluation on the trustllm, or-bench and advbench ood benchmarks demonstrates that prood achieves significantly higher refusal accuracy compared to existing state-of-theart llms, including gpt-4. these results validate our hypothesis that modeling semantic joint distributions yields more precise ood detection, especially for prompts that fall outside the model's epistemic boundaries (e.g., real-time knowledge or multimodal input).