R ain P roof: An Umbrella to Shield Text Generator from Out-Of-Distribution Data (2023)

Abstract: Implementing effective control mechanisms to ensure the proper functioning and security of deployed NLP models, from translation to chatbots, is essential. A key ingredient to ensure safe system behaviour is Out-Of-Distribution (OOD) detection, which aims to detect whether an input sample is statistically far from the training distribution. Although OOD detection is a widely covered topic in classification tasks, most methods rely on hidden features output by the encoder. In this work, we focus on leveraging soft-probabilities in a black-box framework, i.e. we can access the soft-predictions but not the internal states of the model. Our contributions include: (i) RAINPROOF a Relative informAItioN Projection OOD detection framework; and (ii) a more operational evaluation setting for OOD detection. Surprisingly, we find that OOD detection is not necessarily aligned with task-specific measures. The OOD detector may filter out samples well processed by the model and keep samples that are not, leading to weaker performance. Our results show that RAINPROOF provides OOD detection methods more aligned with task-specific performance metrics than traditional OOD detectors.

## introduction significant progress has been made in natural language generation (nlg) in recent years with the development of powerful generic (e.g., gpt @xcite @xcite , llama @xcite and its variants) and task-specific (e.g., grover @xcite , pegasus @xcite and dialog-gpt @xcite )) text generators. they power machine translation (mt) systems or chatbots that are exposed to the public, and their reliability is a prerequisite for adoption. text generators are trained in the context of a so-called closed world @xcite , where training and test data are assumed to be drawn i.i.d. from a single distribution, known as the in-distribution. however, when deployed, these models operate in an open world @xcite where the i.i.d. assumption is often violated. changes in data distribution are detrimental and induce a drop in performance. it is necessary to develop tools to protect models from harmful distribution shifts as it is a clearly unresolved practical problem @xcite . for example, a trained translation model is not expected to be reliable when presented with another language (e.g. a spanish model exposed to catalan, or a dutch model exposed to afrikaans) or unexpected technical language (e.g., a colloquial translation model exposed to rare technical terms from the medical field). they also tend to be released behind api(openai, 2023) ruling out many usual features-based ood detection methods. most work on out-of-distribution (ood) detection focus on classification, leaving ood detection in (conditional) text generation settings mainly unexplored, even though it is among the most exposed applications. existing solutions fall into two categories. the first one called training-aware methods @xcite , modifies the classifier training by exposing the neural network to ood samples during training. the second one, called plug-in methods aims to distinguish regular samples in the in-distribution (in) from ood samples based on the model's behaviour on a new input. plug-in methods include maximum softmax probabilities (msp) @xcite or energy @xcite or featurebased anomaly detectors that compute a per-class anomaly score @xcite @xcite . although plug-in methods from classification settings seem attractive, their adaptation to text generation tasks is more involved. while text generation can be seen as a sequence of classification problems, i.e., chosing the next token at each step, the number of possible tokens is two orders of magnitude higher than usual classification setups. in this work, we aim to develop new tools to build more reliable text generators which can be used in practical systems. to do so, we work under 4 constraints: (i) we do not assume we can access ood samples; (ii) we suppose we are in a blackbox scenario: we do not assume we have access to the internal states of the model but only to the soft probability distributions it outputs; (iii) the detectors should be easy enough to use on top of any existing model to ensure adaptability; (iv) not only should ood detectors be able to filter ood samples, but they also are expected to improve the average performance on the end-task the model has to perform. our contributions. our main contributions can be summarized as follows: ## a more practical evaluation following previous work, we measure the performance of the detectors on the ood detection task based on auroc and fpr ↓. however, this evaluation framework neglects the impact of the detector on the overall system's performance and the downstream task it performs. we identify three main evaluation criteria that are important in practice: (i) execution time, (ii) overall system performance in terms of the quality of the generated answers, and (iii) interpretability of the decision. our study is conducted on nmt because due to the existence of relevant and widely adopted metrics for assessing the quality of a generated sentence (i.e., bleu @xcite and bert-s @xcite and @xcite ). ## rainproof on llm for nmt as an alternative to nmt models, we can study the performance of instruction finetuned llm on translation tasks. however, it is important to note that while llms are trained on enormous amounts of data, they still miss many languages. typically, they are trained on around 100 languages @xcite , this falls far short of the existing 7000 languages. in our test-bed experiments, we decided to rely on bloom models, which have not been specifically trained on german (deu) data. therefore, we can use german samples to simulate ood detection in an instruction-following translation setting, specifically relying on bloomz @xcite . we prompt the model to trans- ## conclusion this work introduces a detection framework called rainproof and a new benchmark called lofter for black-box ood detection on text generator. we adopt an operational perspective by not only considering ood performance but also task-specific metrics: despite the good results obtained in pure ood detection, ood filtering can harm the performance of the final system, as it is the case for a msp or a m . we found that rainproof succeed in removing ood while inducing significant gains in translation performance both on ood samples and in general. in conclusion, this work paves the way for developing text-generation ood detectors and calls for a global evaluation when benchmarking future ood detectors.