R AIN P ROOF : A N UMBRELLA TO SHIELD TEXT GENERATORS FROM O UT -O F -D ISTRIBUTION DATA
Anonymous authors
Paper under double-blind review
A BSTRACT
As more and more conversational and translation systems are deployed in production, it is essential to implement and to develop effective control mechanisms guaranteeing their proper functioning and security.
An essential component to ensure safe system behavior is out-of-distribution (OOD) detection, which aims at detecting whether an input sample is statistically far from the training distribution.
Although OOD detection is a widely covered topic in classification tasks, it has received much less attention in text generation.
This paper addresses the problem of
OOD detection for machine translation and dialog generation from an operational perspective.
Our contributions include: (i) RAINPROOF a Relative informAItioN
Projection ODD detection framework; and (ii) a more operational evaluation setting for OOD detection.
Surprisingly, we find that OOD detection is not necessarily aligned with task-specific measures.
The OOD detector may filter out samples that are well processed by the model and keep samples that are not, leading to weaker performance.
Our results show that RAINPROOF breaks this curse and achieve good results in OOD detection while increasing performance.
I NTRODUCTION
Significant progress have been made in Natural Language Generation (NLG) in recent years with the development of powerful generic ( e.g., GPT (Radford et al., 2018; 2019; Brown et al., 2020)) and task-specific ( e.g., Grover (Zellers et al., 2019), Pegasus (Zhang et al., 2020) and DialogGPT (Zhang et al., 2019)) text generators.
Text generators power machine translation systems or chat bots that are by definition exposed to the public and whose reliability is therefore a prerequisite for adoption.
Text generators are trained in the context of a so-called closed world (Antonucci et al., 2021; Fei & Liu, 2016), where training and test data are assumed to be drawn i.i.d.
from a single distribution, known as the in-distribution.
However, when deployed, these models operate in an open world (Parmar et al., 2021;
Zhou, 2022) where the i.i.d.
assumption is often violated.
This change in data distribution is detrimental and induces a drop in performance as illustrated in Tab. 3 and Tab. 4.
Thus, to ensure the trustworthiness and adoption, it is necessary to develop tools to protect them from harmful distribution shifts.
For example, a trained translation model is not expected to be reliable when presented with another language ( e.g. a Spanish model exposed to Catalan, or a Dutch model exposed to Afrikaans) or unexpected technical language ( e.g. , a colloquial translation model exposed to rare technical terms from the medical field).
Most of the existing research, which aims to protect models from Out-Of-Distribution (OOD) data, focuses on classification. Despite their importance, (conditional) text generation has received much less attention even though it is among the most exposed applications.
Existing solutions fall into two categories.
The first one called training-aware methods (Zhu et al., 2022; Vernekar et al., 2019a;b) modifies the classifier training by exposing the neural network to OOD samples during training.
The second one called plug-in methods aims at distinguishing regular samples in the in distribution (IN) from OOD samples based on the behavior of the model on a new input.
Plug-in methods include
Maximum Softmax Prediction (MSP) (Hendrycks & Gimpel, 2016) or Energy (Lee et al., 2018a) or feature-based anomaly detectors that compute a per-class anomaly score (Ming et al., 2022; Ryu et al., 2017; Huang et al., 2020;
Ren et al., 2021a).
Although plug-in methods seem attractive, their adaptation to text generation may not be straightforward. The sheer number of words present in the vocabulary prevents it to be used directly within the classification framework.
In this work, we aim at developing new tools to build more reliable text generators, which can be used in practical systems.
First, we work in the unsupervised detection setting where we do not assume that we have access to OOD samples as they are often not available.
Second, we work in the black-box scenario , which is the most common in the Software as a Service framework Rudin & Radin (2019). In the black-box setting detection methods only have access to the output of the DNN architecture.
Third, we want an easy-to-use and effective method to ensure adoptability .
Last, we argue that OOD detection impacts on tasks specific performance of the whole system should be taken into account when choosing OOD detectors in an operational setting. Our contributions.
Our main contributions can be summarized as follows:
A more operational benchmark for text generation OOD detection.
We present LOFTER the
L anguage O ut o F dis T ribution p
E rformance benchma R k.
Existing works on OOD detection for language modeling (Arora et al., 2021) focus on (i) english language only, (ii) the GLUE benchmark and (iii) measure performance solely in terms of OOD detection.
LOFTER is, in our view, a more operational setting with a strong focus on neural machine translation (NMT) and dialog generation.
First, it introduces more realistic data shifts that go beyond English Fan et al. (2021): language shifts induced by closely related language pairs ( e.g. , Spanish and Catalan or Dutch and Afrikaans 1 ) and domain change ( e.g. , medical vs news data or different types of dialogs).
In addition, LOFTER comes with an updated evaluation setting: detectors’ performance are jointly evaluated w.r.t the overall system’s performance on the end task. Novel information theoretic-based detectors .
We present
RAINPROOF
a R elative inform AI tio N
P
rojection O ut OF distribution detector
RAINPROOF is
fully
unsupervised .
It
is
flexible and can
be applied both when no reference samples (IN) are available (corresponding to scenario s and when they are (corresponding to scenario s
RAINPROOF tackles s
0 by computing the models’ predictions negentropy (Brillouin, 1953). For s 1 , it relies its natural extension: the Information
Projection (Kullback, 1954; Csisz´ar, 1967), an information-theoretic tool that remains overlooked by the machine learning community.
New insights on the operational value of OOD detectors Our extensive experiments on LOFTER show that OOD detectors may filter out samples that are well processed by the model and keep samples that are not, leading to weaker performance .
Our results show that RAINPROOF breaks this curse and achieve good results in OOD detection while increasing performance.
Code and reproductibility.
After acceptance, we will publish the open-source code on github.
com and the data to facilitate future research, ensure reproducibility and reduce computational costs.
P ROBLEM S TATEMENT & R ELATED W ORKS 2. N OTATIONS & C ONDITIONAL T EXT G ENERATION
Let us denote Ω a vocabulary of size | Ω | and Ω ∗ its Kleene closure (Fletcher et al., 1990) 2 .
We denote P (Ω) =
(cid:110) p ∈ [0 , 1] | Ω | : (cid:80) | Ω | i =1 p
i = 1 (cid:111) the set of probability distributions defined over Ω .
Let D train be the training set, composed of N ⩾ 1 i.i.d.
samples { ( x i , y i ) } Ni =1 ∈ ( X × Y ) N with probability law p XY .
We denote p X and p Y the associated marginal laws of p XY .
Each x i is a sequence of tokens and we denote x ij ∈ Ω the j th token of the i th sequence.
x i ⩽ t = { x i 1 , · · · , x it } ∈ Ω ∗ denotes the prefix of length t .
The same notations hold for y .
Conditional textual generation.
In conditional textual generation, the goal is to model a probability distribution p
⋆ ( x , y ) over variable-length text sequences ( x , y ) by finding p θ ≈ p ⋆ ( x , y ) for any ( x , y ) .
In this work, we assume to have access to a pretrained conditional language model f
θ : X × Y → R | Ω | where the output is the (unormalized) logits scores.
f θ parameterized p θ , i.e., for any ( x , y ) , p θ ( x , y )
=
softmax( f θ ( x , y ) /T )
where T ∈ R denotes the temperature.
Given an input sequence x , the pretrained language f θ can recursively generate an output sequence ˆy by sampling y t +1 ∼ p
Tθ ( ·| x , ˆ y ⩽ t ) , for t ∈ [1 , | y | ] .
Note that ˆ y 0 is the start of sentence ( < SOS > token).
We denote by S ( x ) , the set of normalized logits scores generated by the model when the initial input is x i.e., S ( x )
= { softmax( f θ ( x , ˆ y ⩽ t )) }
| ˆ y | t =1 .
Note that elements of S ( x ) are discrete probability distributions on Ω .
2. P ROBLEM S TATEMENT
In OOD detection the goal is to find an anomaly score a : X → R + that quantifies how much a sample is far from the IN distribution.
x is classified as IN or OUT according to the score a ( x ) .
Following previous work (Hendrycks & Gimpel, 2016), one fixes a threshold γ and classifies the test sample IN if a ( x ) ⩽ γ or OOT if a ( x ) > γ .
Formally, let us denote g ( · , γ ) the decision function, we take: g ( x , γ ) =
[Equation]
Remark 1 .
In our setting, OOD examples are not available .
In our experiments, we take γ such that at least 80% of the train set is classified as IN data.
This assumption is reasonable since, in practice, even a well tailored dataset might contains significant shares of outliers (Mishra et al., 2020).
2. R EVIEW OF OOD DETECTORS
OOD detection for classification.
Most works on OOD detection have focused on detectors for classifiers and relies either on internal representations ( features-based detectors ) or on the final soft probabilities produced by the classifier ( softmax based detectors ).
Features-based detectors. They leverage latent representations to derive anomaly scores (Kirichenko et al., 2020; Zisselman & Tamar, 2020).
The most well-known is the Mahanalobis distance (Lee et al., 2018b; Ren et al., 2021b) but there are other methods employing Grams matrices (Sastry & Oore, 2020), Fisher Rao distance (Gomes et al., 2022) or other statistical tests (Haroush et al., 2021).
Other methods rely on the gradient space (Huang et al., 2021) or the moment of the features (Quintanilha et al., 2019; Sun et al., 2021).
These methods require access to the latent representations of the models, which does not fit the black-box scenario.
Moreover, they often rely on a per-class decision, which is fine for classifiers but the sheer number of words in Ω makes it impossible to use for text generation.
Softmax-based detectors. These detectors rely on the soft probabilities produced by the model.
The maximum softmax probability (Hendrycks & Gimpel, 2017; Hein et al., 2019; Liang et al., 2018; Hsu et al., 2020) uses the probability of the mode while others take into account the entire distribution, such as the Energy-based OOD detection scores (Liu et al., 2020).
Due to the large vocabulary size, it is unclear how these methods generalize to sequence generation tasks.
OOD detection for text generation.
Little work has been done on OOD detection for text generation.
Therefore, we will follow Arora et al.
and will rely on their baselines but also generalize common OOD scores such as MSP or Energy to the context of text generation. Generalization to sequence generation.
We generalize common OOD detectors for classification tasks by computing the average OOD score along the sequence at each step of the text generation.
We refer the reader to Sec.
A.6 for more details.
Remark 2 .
Note that features-based detectors assume a white-box framework where the internal representations of an input are accessible. By contrast to softmax-based detectors which only rely on the final output.
Following Arora et al.
(2021), we work in a black-box framework (Chen et al., 2020).
We also compare our results to the Mahalanobis distance (Lee et al., 2018b), as it is known to be a strong baseline.
RAINPROOF AN I NFORMATION T HEORETIC OOD DETECTORS 3. I NFORMATION T HEORETICAL B ACKGROUND
An information measure I : P (Ω) × P (Ω) → R quantifies the similarity between any pair of discrete distributions p , q ∈ P (Ω) .
Since Ω is a finite set, we will adopt the following notations p =
[ p 1 , · · · , p | Ω | ] and q = [ q 1 , · · · , q | Ω | ] .
The development of new information measures for specific applications has received much attention over the years (Fujisawa & Eguchi, 2008; Cichocki et al., 2011)
(we refer the reader to Basseville (2013) for a complete review).
While there exist information distances, it is, in general, difficult to build metrics that satisfy all the properties of a distance, thus we often rely on divergences which drop the symmetry property and the triangular inequality.
In what follows, we motivate the information measures we will use in this work.
First, we rely on the R´enyi divergences (Csisz´ar, 1967).
R´enyi divergences belong to the f -divergences family and are parametrized by a parameter α ∈ R + −{ 1 } . They are flexible and include well-known divergences such as the Kullback-Leiber divergence (KL) Kullback (1959) (when α → 1 ) or the Hellinger distance (Hellinger, 1909) (when α = 0 . 5 ).
The R´enyi divergence between p and q is defined as follows:
[Equation]
The Renyi divergence is widely used in machine learning (Peters et al., 2019) because α allows weighting the relative influence of the distributions’ tail.
Second, we investigate the Fisher-Rao distance ( FR ).
FR is a distance on the Riemannian space formed by the parametric distributions, using Fisher information matrix as its metric (Amari, 2012).
It computes the geodesic distance between two discrete distributions (Rao, 1992; Pinele et al., 2020) and is defined as follows:
[Equation]
It has recently found many applications (Picot et al., 2022; Colombo et al., 2022b;a) and is known to be more accurate than popular divergence measures (Costa et al., 2015).
3. RAINPROOF FOR THE NO - REFERENCE SCENARIO ( s 0 )
At inference time, the no-reference scenario ( s 0 ) does not assume the existence of a reference set of IN samples to decide whether a new input sample is OOD.
Softmax-based detectors such as MSP (Hendrycks & Gimpel, 2016), Energy (Liu et al., 2020) or the sequence likelihood (Arora et al., 2021) are examples of OOD scores operating under s 0 .
Under these assumptions, our OOD detector RAINPROOF is composed of three steps.
For a given input x with generated sentence
ˆ y :
We first use f θ to extract the step-by-step sequence of soft distributions S ( x ) .
We then compute an anomaly score ( a I ( x ) ) by averaging a step-by-step score provided by I . This step-by-step score is obtained by measuring the similarity between a reference distribution u ∈ P (Ω) and one element of S ( x ) .
Formally:
[Equation]
where |S ( x ) | = | ˆ y | .
The last step consists in thresholding the previous anomaly score a I ( x ) .
If a I ( x ) is over a given threshold γ , we classify x as an OOD example.
Interpretation of Eq.
a I ( x ) measures the average dissimilarity of the probability distribution of the next token to normality (as defined by u ).
a I ( x ) also corresponds to the token average uncertainty of the model f θ to generate ˆy when the input is x .
The intuition behind Eq.
3 is that the distributions produced by f θ , when exposed to an OOD sample, should be far from normality and thus should have a high score.
Choice of u and I .
The uncertainty definition of Eq.
3 depends on the choice of both the reference distribution u and the information measure I .
A natural choice for u is the uniform distribution, i.e., u = [ 1 | Ω | , · · · , 1 | Ω | ] which we will use in this work.
It is worth pointing out that I ( ·|| u ) yields the negentropy of a distribution.
Other possible choices for u include one hot or tf-idf distribution (Colombo et al., 2022b).
For I , we rely on the R´enyi divergence to obtain a D α and the Fisher-Rao distance to obtain a FR .
3. RAINPROOF FOR THE REFERENCE SCENARIO ( s 1 )
In the with reference scenario ( s 1 ), we assume that one has access to a reference set of IN samples R = { x i : ( x i , y i ) ∈ D train } |R| i =1 where |R| is the size of the reference set.
For example, the Mahalanobis distance works under this assumption.
One of the weakness of Eq. 3 is to imposes an ad-hoc choice when using u (the uniform distribution).
In s 1 , we can leverage R , to obtain a data-driven notion normality .
Under s 1 , our OOD detecor RAINPROOF follows these four steps:
( Offline )
For each x i ∈ R , we generate ˆ y i and the associated sequence of probability distributions ( S ( x i ) ).
Overall we thus generate (cid:80) x ∈R | ˆy i | probability distributions which
could explode for long sequences.
To overcome this limitation, we rely on the bag of distributions of each sequence (Colombo et al., 2022b).
We form the set of these bags of distributions
[Equation]
( Online )
For a given input x with generated sentence ˆ y , we compute its bag of distributions representation
[Equation]
( Online )
For x , we then compute an anomaly score a ⋆ I ( x ) by projecting ¯p ( x ) on the set ¯ S ∗ .
Formally, a ⋆ I ( x ) is defined as:
[Equation]
We denote p ⋆ ( x ) = arg min [Equation] The last step consists in thresholding the previous anomaly score a I ( x ) .
If a I ( x ) is over a given threshold γ , we classify x as an OOD example.
Interpretation of Eq.
a I ( x ) relies on a Generalized Information Projection (Kullback, 1954; Csisz ´ ar, 1975; 1984) 3 which measures the similarity between ¯p ( x ) and the set ¯ S ∗ .
Note that the closest element of ¯ S ∗ in the sens of I can give insights on the decision of the detector. It allows to interpret the decision of the detector as we will see in Tab. 5.
Choice of I .
Similarly to Sec. 3.2, we will rely on the R ´ enyi divergence to define a ⋆ R α ( x ) and the
Fisher-Rao distance a ⋆ FR ( x ) .
R ESULTS ON LOFTER 4. LOFTER : L ANGUAGE O UT O F DIS T RIBUTION P E RFORMANCE BENCHMA R K
LOFTER for NMT.
We consider two main types of changes: language changes and domain changes, which both can occur in real-world situations.
For each shift, we rely on pretrained generators from the HuggingFace Hub.
Further experiemental details are relegated to Ap.
A. Language shifts can appear when a translation system is exposed to a language that is extremely similar to the language the system has been trained on ( e.g. , Afrikaans for a system trained on Dutch) and, therefore, can lead to significant translation errors (see Tab. 7)).
For language shifts, we focus on closely related language pairs coming from the Tatoeba dataset (Tiedemann, 2012b) (see Tab. 6).
We study the shifts induced by Catalan-Spanish, Portugese-Spanish and Afrikaans-Dutch.
Domain shifts , which occur when the model is exposed to a specific topic that was not seen during training, can also affect the quality of the translation (see Tab. 4).
To simulate domain shifts, we use the language Tatoeba MT dataset (Tiedemann, 2020) and the news commentary dataset (Tiedemann, 2012b) as base datasets
[Table]
and the shifts are induced by the EuroParl dataset (Tiedemann, 2012a) and EMEA (Tiedemann, 2012b) dataset.
LOFTER for dialogs.
For conversational agents, an interesting scenario is when a goal-oriented agent designed to handle a specific type of conversations ( e.g. , customer conversations, daily dialogue) is exposed to an unexpected conversation.
In this case, it is crucial to interrupt the agent so it does not damage the user’s trust with misplaced responses (Perez et al., 2022).
We rely on the Multi WOZ dataset (Zang et al., 2020), a human to human dataset collected in the Wizard-of-Oz set-up (Kelley, 1984), for IN distribution data.
This choice is mostly motivated by the availability of pretrained models on Multi WOZ .
For dialog shifts, we use spoken datasets coming from various sources which are part of the SILICONE benchmark (Chapuis et al., 2020).
Specifically, we use a goaloriented dataset ( i.e. , Switchboard Dialog Act Corpus ( SwDA ) (Stolcke et al., 2000)), a multi-party meetings dataset ( i.e. , MRDA (Shriberg et al., 2004) and Multimodal EmotionLines Dataset MELD (Poria et al., 2018)), daily communication dialogs ( i.e. , DailyDialog DyDA
Li et al.
(2017)), and scripted scenarii ( i.e. , IEMOCAP Tripathi et al.
We refer the curious reader to Sec.
A.4 for more details on each dataset.
Metrics.
OOD detection is usually framed as an unbalanced binary classification problem where the class of interest is OUT.
We can assess the performance of our OOD detectors focusing on the False alarm rate (FPR) and on the True detection rate (TPR).
To evaluate the performance on the OOD task we report the AUROC ↑ and the FPR ↓ .
Area Under the Receiver Operating Characteristic curve ( AUROC ↑ ) (Bradley, 1997).
The AUROC ↑ can be interpreted as the probability that an IN-distribution example has an higher anomaly score than an OOD sample.
For this metric, higher is better.
False Positive Rate at r % True Positive Rate ( FPR ↓ ) .
In many practical application, we have to detect at least r % of the the OOD samples. This corresponds to pre-defined safety level. FPR ↓ quantifies the share of IN samples we wrongly detect under this constraint.
It leads to select a threshold γ r such that the corresponding TPR equals r .
In our work r is set to 95% . Additional details on these metrics can be found in Sec.
A.1.
4. E XPERIMENTS IN MACHINE TRANSLATION AND RESULTS
Results on language shifts.
We assess, for each language pair, the OOD detection performance of RAINPROOF and report the average AUROC ↑ and FPR ↓ in Tab. 1a.
We provide the detailed results in Tab. 8.
We find that our no-reference methods ( a D α and a FR ) achieve better performance that common no-reference baselines but also outperform the reference-based baseline.
In particular, a D α , by achieving an AUROC ↑ of 0 .
95 and FPR ↓ of 0 .
25 , outperforms all considered methods. Moreover, while no-reference baselines only capture up to 62% of the OOD samples on average, ours detect up to 83 .
5% , achieving even better results than the with-reference baseline ( 75 . 3% ).
Results on domain shifts.
We evaluate the OOD detection performance of RAINPROOF on domain shifts in Spanish and German with technical medical data and parliamentary data. We report the average OOD detection performance in Tab. 1a. In s 0 , we observe that a D α and a FR outperform the strongest baselines ( i.e. , Energy, MSP and sequence likelihood) by several AUROC ↑ points. Interestingly enough even our no-reference detectors outperform the reference-based baseline ( i.e. , a M ).
However, we find that relying on a reference set is a must-have in terms of FPR ↓ . While a D α achieves similar AUROC ↑ performance to its information projection counterpart a D ∗ α , the latter achieve much better FPR ↓ . 4. E XPERIMENTS IN DIALOG GENERATION AND RESULTS
[Table]
Results on Dialog shifts.
The dialog shifts benchmark is more difficult than NMT benchmark as all detectors achieve lower performances. It is the only case where our no-reference detectors do not outperform the Mahalanobis baseline and achieve only 0 .in AUROC ↑ .
The best baseline is the Mahalanobis distance and achieves better performance on dialog task than on NMT domain shifts reaching an AUROC ↑ of 0 . However, our reference based detector based on the
R´enyi information projection secures better AUROC ↑ ( 0 . 86 ) and better FPR ↓ ( 0 . 52 ). Even though RAINPROOF outperforms all the baselines, shifts in dialog are hard to detect and will require further investigations.
Non-aggregated results for dialog are provided in Ap. C. They show that RAINPROOF consistently outperforms baselines on all datasets. [Figure]
Importance of distribution tails.
Our results show that, when it comes to domain shift (domain shifts in translation or dialog shifts), reference-based detectors are required to obtain good results.
They also show that, the more these detectors take into account the tail of the distributions, the better they are, as displayed in Sec. B.1.
We find that low values of α (near 0 ) yields better results with the R ´ enyi Information projection a D ∗ α .
It suggests that the tail of the distributions used during text generation carries context information and insights on the processed texts.
Such results are consistent with findings of recent works in the context of automatic evaluation of text generation (Colombo et al., 2022b).
Comparison to the Mahalanobis distance Our reference-based detector work with a small reference set.
In our experiments, we use reference sets of size 10 to 2000 .
The Mahalanobis distance requires to approximate the covariance matrix of the reference set.
In our simulations, the embeddings of dimension 512 make the estimation unreliable.
On the contrary, RAINPROOF , which rely on information projections, remains numerically sound with small reference set.  T OWARDS A PRACTICAL EVALUATION OF OOD DETECTORS Following previous work, we measure the performance of the detectors on the OOD detection task based on AUROC ↑ and FPR ↓ . However, this evaluation framework neglects the impact of the detector on the overall system’s performance.
We identify three main evaluation criteria that are important in practice: execution time, overall system performance in terms of quality of the generated sentences, and interpretability of the decision. Our study is conducted on NMT because due to the existence of relevant and widely adopted metrics for assessing the quality of a generated sentence ( i.e. , BLEU (Papineni et al., 2002) and BERTSCORE ( BERT-S ) (Unanue et al., 2021)). [Table] 5. C OMPLEXITY S TUDY
Runtime and memory costs. We report in Tab. 1b the runtime of all methods.
Detectors for s 0 are faster than the ones for s 1 . Contrarily to detectors using references, the no-reference detectors do not require additional memory. They can be setup easily in a plug&play manner at the output of any model. Numerical stability.
The Mahalanobis distance requires to estimate both µ and Σ − 1 (see Sec. A.6). The dimension of the latent space of the considered pre-trained model is either 768 or 512. In this setting, when the size of the reference set is small, the estimation of the Mahalanobis parameters is numerically unstable. For s 1 , RAINPROOF relies on information projection and does not involve numerically unstable computations but requires a larger memory footprint (0.5 GB) to store the reference set (2000 probability distributions of dimension 50K).
5. I MPACT OF OOD FILTERING ON TRANSLATION QUALITY
The main objective of OOD filtering is to remove samples that are far from the training distribution. On these samples, the user has no guarantee that the model will produce a good quality translation.
In this experiment, we compare the performance of the system with and without the different detectors in terms of the quality of the generated sentence. Global performance. In Tab. 3, we report the global performance of the systems ( f θ ) without and with OOD detectors on IN samples, OOD samples and all samples (ALL). From the first row of
Tab. 3, we notice that OOD samples are harmful to the model. We observe that, in most of the cases, adding detectors increases the model performance on IN, OOD and all samples. Exceptions include a MSP (for OOD, IN and ALL) and a M (for OOD). Results indicate that no-reference RAINPROOF outperforms the reference-based version of RAINPROOF .
Thus, OOD detector evaluation should consider the final task performance.
Overall, it is worth noting that directly adapting classical OOD detection methods ( e.g. , MSP or Energy) to the sequence generation problem leads to poor results in terms of performance gains ( i.e. , as measured by BLEU or BERT-S ). In others words, the final task does not benefit from adding classical OOD detectors. Finer performance analysis.
In Tab. 4, we report the per-shift-types performance of f θ with and without OOD detector. In Tab. 4, we observe a decrease in performance in the case of language and domain shifts, the latter being more harmful. On domain shifts, we observe that reference-based detectors decrease system’s performance on OOD samples.
This means that the detectors tend to filter out samples that are well-handled by the model and ignore sentences that are not. It is worth noting that reference-based detectors remove, in proportion, twice as many samples as their no-reference counterparts, while the threshold selection procedure remains the same. This observation also holds when removing less samples ( i.e. , calibrating γ that we remove 10%, 5% or even 1% of the IN dataset) (Tab. 15). [Table]
Threshold free analysis.
In Tab. 2, we report the correlation between OOD scores and final task performance for the case of domain shifts.
We refer the reader to Tab. 14 for the results on language shifts.
We observe that the likelihood score is the most correlated with the final sentence quality, as measured by BLEU or BERT-S .
This finding illustrates that higher correlation with sentence quality does not necessarily translate into higher performance gains when filtering OOD samples.
This result suggests that Quality Estimation (Specia et al., 2010; Blatz et al., 2004) , while closely related, is a different problem.
5. T OWARDS AN INTERPRETABLE DECISION
The first 2 are far from the reference set and not well translated whereas the next 2 are very close to the reference set and well translated.
We can, for that matter, notice that the projection is quite close to the input sentence grammatically speaking.
[Table]
An important dimension fostering adoption is the ability to verify the decision taken by the automatic system (Montavon et al., 2018).
RAINPROOF offers a step in this direction when used with references: for each input sample, RAINPROOF finds the closest sample (in the sens of the Information Projection) in the reference set to take its decision.
We present in Tab.
5 some OOD samples along with their translation scores, projection scores
[Table]
and their projection on the reference set.
We notice that, in general, sentences that are close to the reference set [Table] and whose projection has a close meaning, are better handled by f θ . Therefore, one can visually interpret the prediction of RAINPROOF , and validate it.
This observation further validate our method.  C ONCLUSIONS In this work, we introduced both a detection framework called RAINPROOF as well as a new benchmark called
LOFTER for detecting OOD samples when using textual generators in the black-box scenario. Our work adopts an operational perspective by not only considering OOD performance but also task-specific metrics. Our results show that, despite the good results obtained in pure OOD detection, OOD filtering can harm the performance of the final system, as it is the case for MSP or Mahanalobis.
We found that, RAINPROOF breaks this curse and induces significant gains in translation performance both on OOD samples and in general. In conclusion, this work paves the way to the development of detectors tailored for text generators and calls for a global evaluation when benchmarking future OOD detectors.
A PPENDIX A Experimental setting
A.1 Additionnal Details on Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.2 Language pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
A.3 Samples . . A.4 Dialog datasets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.5 Choices of models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . A.6 Generalization of existing OOD detectors to Sequence Generation . . . . . . . . .
B Parameters tuning
B.1 Impact of α . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C Performance of our detectors in OOD detection
C.1 Summary of our results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
C.2 Detailed results of OOD detection performances . . . . . . . . . . . . . . . . . . . C.3 ROC AUC curves . . .
D NTM performance
D.1 Absolute
performances
D.2 Gains . . D.3 Effect of a Larger threshold on NMT performance . . . . . . . . . . . . . . . . . .
E Negative results
E.2 Negentropy of bag of distributions
A E XPERIMENTAL SETTING
In this section we dive into the details and definitions of our experimental setting.
First we present our OOD detection performance metrics (Sec. A.1), then we provide a couple samples for one of the small language shifts (Sec. A.3).
We also discusse the choices of pretrained model (Sec. A.5) and how we adapted common OOD detectors to the text generation case (Sec. A.6).
A. A DDITIONNAL D ETAILS ON M ETRICS
OOD Detection is usually an unbalanced binary classification problem where the class of interest is OUT.
Let us denote Z the random variable corresponding to actually being out of distribution.
We can assess the performance of our OOD detectors focusing on the False alarm rate and on the True detection rate . The False alarm rate or False positive rate (FPR) is the proportion of samples missclassified as OUT. For a score threshold γ , we have FPR =
Pr (cid:0) a ( x ) > γ
| Z = 0 (cid:1) .
The True detection rate or True positive rate (TPR) is the proportion of OOD samples that are detected by the method.
It is given by TPR =
Pr (cid:0) a ( x ) > γ | Z = 1 (cid:1) .
In order to evaluate the performance of our methods we will focus and report mainly the AUROC ↑ and the FPR ↓ , we provide more detailed metrics and experiments in Sec.
A.1.
Area Under the Receiver Operating Characteristic curve ( AUROC ↑ ) Bradley (1997).
The
Receiver Operating Characteristic curve is curve obtained by plotting the True positive rate against the False positive rate.
The area under this curve is the probability that an in-distribution example
X in has a anomaly score higher than an OOD sample x out : AUROC ↑ =
Pr( a ( x in ) > a ( x out )) .
It is given by γ (cid:55)→ (Pr (cid:0) a ( x ) > γ | Z = 0 (cid:1) , Pr (cid:0) a ( x ) > γ | Z = 1 (cid:1) ) .
False Positive Rate at 95% True Positive Rate ( FPR ↓ ) .
We accept to allow only a given false positive rate r corresponding to a defined level of safety and we want to know what share of positive samples we actually catch under this constraint.
It leads to select a threshold γ r such that the corresponding TPR equals r .
At this threshold, one then computes: Pr( a ( x ) > γ r | Z = 0) with γ r s.t.
TPR ( γ r ) = r .
r is chosen depending of the difficulty of task at hand and the required level of safety.
For the sake of brevity we present only AUROC ↑ and FPR ↓ metrics in our aggregated results but we also used Detection error and Area Under the Precision-Recall curve metrics and those are presented in our full results section (Ap. C).
Detection error .
It is simply the probability of miss-classification for a given True positive rate.
Area Under the Precision-Recall curve ( AUPR-IN / AUPR-OUT ) Davis & Goadrich (2006) .
The
Precision-Recall curve plots the recall (true detection rate) against the precision (actual proportion of OOD amongst the predicted OOD).
The area under this curve γ (cid:55)→ (Pr (cid:0) Z = 1 | s ( X ) ⩽ γ (cid:1) , Pr (cid:0) s ( X ) ⩽ γ | Z = 1 (cid:1) )
captures the trade-off between precision and recall made by the model.
A high value represents a high precision and a high recall i.e. the detector captures most of the positive samples while having few False positive. A.  [Table] ES
Eurparl DE [Table] Ellm’haditthatthesevahousewashaunted. 27.78 Aquest ´esellloconvan`eixerelmeupare.
Thisistheplacewheremyfatherwasborn.
Aquestisthelloconvan`eixerelmeupare.
A. D IALOG DATASETS
Switchboard Dialog Act Corpus ( SwDA ) is a corpus of telephonic conversations.
The corpus provides labels, topic and speaker information (Stolcke et al., 2000).
ICSI MRDA Corpus ( MRDA ) contains transcript 75 h of naturally occuring meetings involving more than 50 people (Shriberg et al., 2004).
DaylyDialog Act Corpus ( DyDA ) contains daily common communications between people, covering topic such as small talk, meteo or daily activities (Li et al., 2017).
Interactive Emotional Dyadic Motion Capture IEMOCAP)
(Tripathi et al., 2018) consists of transcripts of improvisations or scripted scenarii supposed to outline the expression of emotions.
A. C HOICES OF MODELS
A lot of pretrained model for conditional text generation are available.
To perform our experiments we needed models that were already well installed and deployed and that would also support OOD settings.
For translation tasks we needed specialized models for a notion of OOD to be easily defined.
It would be indeed more hazardous to define a notion of OOD language when working with a multilingual model.
The same is true for conversational models.
Neural Machine Translation model We benchmark our OOD method on translation models provided by Helsinky NLP Tiedemann & Thottingal (2020) on several pairs of languages with large and small shifts.
We extended the experiment to detect domain shifts.
These models are indeed specialized in each language pairs and are widely recognize in the neural machine translation field.
Conversational model We used a dialogGPT Zhang et al.
(2019) model fine-tuned on the Multi
WOZ dataset as chat bot model.
The finetuning on daily dialogue type tasks ensure that the model is specialized, thus allowing us to get a good definition of samples not being in its range of expertise.
Moreover, the choice of the architure, DialogGPT, guarantee that our results are valid on a very common architure.
A. G ENERALIZATION OF EXISTING OOD DETECTORS TO S EQUENCE G ENERATION
In this section, we extend classical OOD detection score to the conditional text generation settting.
Common OOD detectors were built for classification tasks and we need to adapt them to conditional text generation.
Our task can be viewed as a sequence of classification problems with a very large number of classes (the size of the vocabulary).
We chose the most naive approach which consists of averaging the OOD scores over the sequence.
We experimented with other aggregation such as the min/max or the standard deviation without getting interesting results.
Likelihood Score
The most naive approach to build a OOD score is to rely solely on the loglikelyhood of the sequence.
For a conditioning x we define the log-likelyhood score by a L ( x ) =
[Equation]
Average Maximum Softmax Probability score The maximum softmax probability Hendrycks & Gimpel (2017) takes the probability of the mode of the categorical distribution as score of
OOD.
We extend thise definition in the case of sequence of probability distribution by averaging this score along the sequence.
For a given conditioning x , we define the average MSP score a MSP ( x ) = 1 | ˆ y | (cid:80) | ˆ y | t =1 max i ∈ [ | 0 ,K | ]
p Tθ ( i | x , ˆ y ⩽ t )) .
While it is closely linked to uncertainty measures it discards most of the information contained in the probability distribution.
It discards the whole probability distribution.
We claim that much more information can be retrieve by studying the whole distribution.
Average Energy score We extend the definition of the energy score described in Liu et al.
to a sequence of probability distributions by averaging the score along the sequence.
For a given conditioning x and a temperature T we define the average energy of the sequence: a E ( x ) ≜ − T | ˆ y | (cid:80)
| ˆ y | t =1 log (cid:80) | Ω | i e f θ ( x , ˆ y ⩽ t ) i /T .
It corresponds to the normalization term of the softmax function applied on the logits.
While it takes into account the whole distribution, it only takes into account the amount of unormalized mass before normalization without attention to how this mass is distributed along the features.
Mahalanobis distance Following Lee et al.
(2018c) compute the Mahalanobis matrice based on the samples of a given reference set R .
In our case we are using encoder-decoder models we use the output of the last hidden layer of the encoder as embedding.
Let’s denote ϕ ( x ) this embedding for a conditionning x . Let’s µ and Σ be respectively the mean and the covariance of these embedding on the reference set. We define a M ( x ) = (cid:0) 1 + ( ϕ ( x ) − µ ) ⊤ Σ − 1 ( ϕ ( x ) − µ ) (cid:1) − 1 .
B P ARAMETERS TUNING
Detectors depend on their anomaly score to make decision and these scores can be parametric. First of all, soft probability based scores depend on the soft probability distribution and its scaling, therefore the temperature is a crucial parameter to tune to get the most performance. While a small temperature tend to make the distribution more picky, higher value spread the probability mass along the classes. Moreover, the renyi divergence and its related informaiton projection depend on a factor α .
We provide here further results and analysis of those parameters on our results.
Indeed, in Fig.
1 we present the impact of the size of the reference set and of the paramet α on Renyi information projection to distinguish dialog shifts, as expected, the larger the reference set, the better.
However, we see that smaller values of α yield better results.
We recall that the Renyi divergence is defined as D α ( p ∥ q ) =
[Equation]
(cid:17) , where α ∈
R + − { 1 } .
Smaller values of α distribute the weight of each feature more equally in the final divergence, more specifically they tend to give an equal weight to the very likely outcome as well as to the less likely ones, therefore giving more weight to the tail of the distribution.
When α tends tothe Renyi divergence actually counts the number of nonzero common probabilities.
That makes sens in terms of topic detection, it counts the common tokens considered during text generation.
C P ERFORMANCE OF OUR DETECTORS IN OOD DETECTION C. S UMMARY OF OUR RESULTS
In Fig.
2 we present the different level of performance of all the detectors we studied.
We can see that in every task our detectors outperform the baselines but also that in dialog shift, while the
Mahalanobis distance outperform clearly our detectors for s 0 , they still outperform baselines for their scenario by far.
[Figure]
[Figure]
[Table]
[Figure]
[Table]
[Figure] [Table] [Figure] C. D ETAILED RESULTS OF OOD DETECTION PERFORMANCES
In this section we present the performances of our OOD detectors on each detailed tasks, i.e. for each pair of IN and OOD data with all the considered metrics.
We show that our metrics outperform other
OOD detectors baselines in almost all scenarios.
[Table]
[Figure]
[Table]
[Table]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
[Figure]
D NTM PERFORMANCE
Surprisingly we show that common OOD detectors tend to exclude samples that are well handled by the model and keep some that are not leading to decreasing overall performance in terms of translation metrics.
Moreoever it seems this phenomenon is more dominant in reference based detectors.
We show that our uncertainty based detectors mostly avoir that downfall and provide good OOD detection and improved translation performances
[Figure]
D.
A BSOLUTE
PERFORMANCES
It is clear (somewhat expected) that NMT models do not perform as well on OOD data as we can see in Tab. 11b.
However, we find that our OOD detectors are able to remove most of the worst case samples and keep enough well translated samples so that with correct filtering our method actually allow the model to achieve somewhat acceptable BLEU scores. [Figure] [Figure] [Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Figure]
[Table]
[Table]
+3.
+
[Table]
+
[Table]
+
[Table]
[Table]
.37 a
[Table]
+
[Table]
[Table]
+
[Table] [Table] + [Table]
+
[Table]
[Table]
+
[Table]
+
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
[Table]
+
[Table]
+ [Table] [Table] [Table]
D. G AINS D. E FFECT OF A L ARGER THRESHOLD ON NMT PERFORMANCE
[Table]
E N EGATIVE RESULTS E. D IFFERENT AGGREGATION OF OOD METRICS
[Table]
Most of our detectors are initially classification
OOD detectors that we adapted for text generation by averaging them over the generated sequences and using this aggregated score as a score for the whole sequence.
We experimented with other aggregations such as the standard deviation or the min/max along the sequence.
If the standard deviation gave relatively good results they were still less interesting that the naive average.
E. N EGENTROPY OF BAG OF DISTRIBUTIONS
We introduced in Sec.
3.3 the bag of distributions as a way to aggregate a sequence of probability distribution and compare it to a set of reference using information projections Sec. 3.3.
A natural idea would be to apply the Negentropy methods (Sec. 3.2) to these aggregated distributions.
More formally given a sequence of probability distribution S θ ( x ) = { p Tθ ( x , ˆ y ⩽ t ) } nt
=1 we would compute its bag of distributions:
[Equation]
And then compute as novelty score:
[Equation]
Further experiments have shown that this process was unable to discriminate OOD samples or improve performance translation.
We suspect that the uncertainty at each step is key to capture the behavior of the language model and that this uncertainty information is lost when averaging probability distribution along the sequence.