CED : Comparing Embedding Differences for Detecting Out-of-Distribution and Hallucinated Text (2024)

Abstract: Detecting out-of-distribution (OOD) samples is crucial for ensuring the safety and robustness of models deployed in real-world scenarios. While most studies on OOD detection focus on fine-tuned models trained on in-distribution (ID) data, detecting OOD in pre-trained models is also important due to computational limitations and the widespread use of open-source pre-trained models. However, in the same domain shift setting, the OOD detection performance of pre-trained models is insufficient because both ID and OOD samples originate from the same domain, leading to a high overlap in their embeddings. To address this issue, we introduce a new method called CED, a training-free OOD detection technique designed to enhance the distinction between ID and OOD datasets. We theoretically validate that specific auxiliary and oracle samples that satisfy certain conditions improve this distinction. Motivated by our theoretical analysis, CED enhances the differentiation by utilizing these specially designed auxiliary and oracle samples. As a result, CED significantly improves the ability of pre-trained models to distinguish between ID and OOD samples in text classification and hallucination detection tasks. Furthermore, we verify that CED is a plug-and-play method compatible with various backbone networks, such as RoBERTa, Llama, and OpenAI Embedding.

## introduction language models trained on large-scale datasets with extensive parameters, known as pre-trained language models (plms), are renowned for their generalization abilities. consequently, a wide range of research has focused on improving plm architectures, such as roberta @xcite , llama @xcite , and gpt3 @xcite . with the advancement of plms, their applications expanded across various fields, demonstrating superior performance on a variety of discriminative and generative tasks, including classification and text completion. given the exceptional performance of recent plms, there is growing interest in deploying them for real-world applications. ensuring reliability in these contexts requires the ability to distinguish between in-distribution (id) and out-of-distribution (ood) datasets, as test distributions may shift dynamically. this capability becomes especially important in critical sectors like healthcare and customer services, where the stakes are high. ## method our proposed method ced consists of two main steps: 1) oracle and auxiliary sample selection, and 2) sample combination and scoring. figure 2 provides an overview of the entire methodology. ## conclusion with the recent advances and wide application of plms, the reliability of plm on diverse tasks is crucial. however, current plms are vulnerable to detecting ood classes or hallucinated text. in this study, we propose a novel ood detection method, ced, that effectively harnesses the inherent relational patterns in pre-trained models. first, we present theoretical findings that certain combinations of auxiliary and oracle samples enhance ood detection performance. especially, we verify that ced, with a certain sample selection strategy, captures the hard ood dataset. second, we implement these theoretical findings into a training-free method called ced. third, we extensively verify the superiority of ced on both discriminative and generative tasks across diverse datasets. our method is plug-and-play and compatible with various ood detection methods and plm backbones.