Out-of-Distribution Detection and Selective Generation for Conditional Language Models
Abstract
Much work has shown that high-performing ML classiﬁers can degrade signiﬁcantly and provide overly-conﬁdent, wrong classiﬁcation predictions, particularly for out-of-distribution (OOD) inputs.
Conditional language models (CLMs) are predominantly trained to classify the next token in an output sequence, and may suffer even worse degradation on out-of-distribution (OOD) inputs as the prediction is done auto-regressively over many steps.
We present a highly accurate and lightweight OOD detection method for CLMs, and demonstrate its effectiveness on abstractive summarization and translation.
We also show how our method can be used under the common and realistic setting of distribution shift for selective generation of high-quality outputs, while automatically abstaining from low-quality ones, enabling safer deployment of generative language models.
Introduction
Recent progress in generative language models (Wu et al., 2016; Radford et al., 2019; Lewis et al., 2020; Raffel et al., 2020; Zhang et al., 2020) has led to quality approaching human-performance on research datasets and has opened up the possibility of their wide deployment beyond the academic setting.
In realistic user-facing scenarios such as text summarization and translation, user provided inputs might signiﬁcantly deviate from the training data distribution.
This violates the independent, identically-distributed (IID) assumption commonly used in evaluating machine learning models.
Many have shown that performance of machine learning models can degrade signiﬁcantly and in surprising ways on OOD inputs (Nguyen et al., 2014; Goodfellow et al., 2014; Ovadia et al., 2019).
This has led to active research on OOD detection for a variety of domains, including vision and text but focused primarily on classiﬁcation.
Conditional language models are typically trained given input sequence to auto-regressively generate the next token in a sequence.
Consequently, the perils of out-of-distribution are arguably more severe as errors propagate and magnify through auto-regression.
Common errors from text generation models include disﬂuencies (Holtzman et al., 2020) and factual inaccuracies (Goodrich et al., 2019; Maynez et al., 2020).
In this work, we propose OOD detection methods for CLMs using abstractive summarization and translation as case studies.
We show that CLMs have untrustworthy likelihood estimation on OOD examples, making perplexity a poor choice for OOD detection.
We propose a highly-accurate, simple, and lightweight OOD score based on the model’s input and output representations (or embeddings) to detect OOD examples, requiring negligible additional compute beyond the model itself.
While accurate OOD detection enables the conservative option of abstaining from generation on
OOD examples, it may be desirable to generate on near-domain data.
Thus the ability to selectively generate where the model is more likely to produce higher-quality outputs, enables safer deployment of conditional language models.
We call this procedure selective generation , analogous to the commonly used term selective prediction in classiﬁcation (Chow, 1957; Bartlett & Wegkamp, 2008; Geifman & El-Yaniv, 2017).
We show that while model perplexity is a reasonable choice for indomain examples, combining with our OOD score works much better when the input distribution is shifted.
OOD Detection in Conditional Language Models
The maximum softmax probability (MSP), p ( y | x ) , y = arg max k =1 ,...,K p ( k | x ) is a simple, commonly used OOD score for K -class classiﬁcation problem (Hendrycks & Gimpel, 2016).
For CLMs, the perplexity, which is monotonically related to the negative log-likelihood of the output sequence averaged over tokens − 1 T (cid:80) Tt =1 log p ( y t | y <t , x ) is a natural OOD score to consider, and analogous to the negative MSP in classiﬁcation.
However, we found that the perplexity distributions overlap signiﬁcantly with each other even though the input documents are signiﬁcantly different, suggesting that perplexity is not well suited for OOD detection (see Figure A.1 for details).
Detecting OOD using CLM’s embeddings We use Transformer encoder-decoder models and obtain the input embedding z by averaging the encoder’s ﬁnal-layer hidden state vectors
h i corresponding to the input sequence token x i .
To obtain the output embedding w we average the decoder’s ﬁnal-layer hidden state vectors
g i corresponding to the output token y i .
See Figure A.2.
Intuitively, if the embedding of a test input or output is far from the embedding distribution of the training data, it is more likely to be OOD.
One way of measuring this distance is to ﬁt a Gaussian,
N ( µ , Σ ) , to the training embeddings and use the Mahalanobis distance (MD) :
[Equation]
This has been used for OOD detection using the representations from classiﬁcation models (Lee et al., 2018) and computing the distances to class-conditional Gaussians.
Unlike classiﬁcation, which has class labels, in CLMs we have paired input and output text sequences.
We ﬁt one Gaussian on the training input embeddings, N ( µ z , Σ z ) , and a second Gaussian on the embeddings of the training ground-truth outputs, N ( µ w , Σ w ) .
For a test input and output embedding pair ( z test , w test ) , the input and output MD are computed as
[Equation]
Ren et al. and Ren et al.
(2021) showed that MD can be confounded by background effect for
OOD in classiﬁcation.
In this work, we extend the idea to CLMs and propose,
[Equation]
where MD 0 ( z test ) :
=
MD ( z test ; µ z 0 , Σ z 0 ) is the MD to a background Gaussian N ( µ z 0 , Σ z 0 ) , ﬁt using a large, broad dataset to approximately represent all domains.
Similarly we deﬁne,
[Equation]
where MD δ ( w test ) :=
MD ( w test ; µ w δ , Σ w δ ) is the MD to the decoded output background distribution
N ( µ w δ , Σ w δ ) .
See Algorithm 1 and 2 for the details.
The RMD score can be regarded as a background contrastive score that indicates how close the test example is to the in-domain compared to the background domain.
We also consider a binary classiﬁer between in-domain and background domain and use the logit as the OOD score, but we found RMD is better at distinguishing near-OOD from far-OOD than binary logits (See Section A.1.2).
Experiments Experiment setup
For summarization, we use a PEGASUS LARGE model (Zhang et al., 2020) ﬁne-tuned on the xsum (Narayan et al., 2018), consisting of BBC News articles with short, abstractive summaries.
We use 10 , 000 examples from xsum and C4 (Raffel et al., 2020) training split to ﬁt [Table] in-domain/foreground and background Gaussian distributions, respectively.
For test datasets, we have cnn_dailymail (Hermann et al., 2015; See et al., 2017), news articles and summaries from CNN and DailyMail; newsroom (Grusky et al., 2018), from 38 major news publications; reddit_
[Table]
(Kim et al., 2018), informal stories from sub-reddit TIFU with author written summaries; samsum (Gliwa et al., 2019) and forumsum (Khalman et al., 2021), high-quality summaries of casual dialogues.
For translation, we train a Transformer base model (Vaswani et al., 2017) on WMT15 En-Fr (Bojar et al., 2015).
We use 100K examples from WMT15 En-Fr and 100K from ParaCrawl En-Fr (Bañón et al., 2020) to ﬁt the foreground and background Gaussian distributions, respectively.
For test, we use newstest2014 ( nt14 ), newsdiscussdev2015 ( ndd15 ), and newsdiscusstest2015 ( ndt15 ) from WMT15 (Bojar et al., 2015), the law , Koran , medical , IT , and subtitles ( sub ) domains from OPUS (Tiedemann, 2012; Aulamo & Tiedemann, 2019), and MTNT (Michel & Neubig, 2018) consisting of noisy comments from Reddit.
3.1 OOD detection Baseline methods
We compare our proposed OOD scores with various baseline methods, including (1) perplexity score, (2) embedding-based Mahalanobis distance, (3) Natural Language Inference (NLI) score (Honovich et al., 2022) for summarization task, and (4) COMET (Rei et al., 2020) and (5) Prism (Thompson & Post, 2020) for translation task.
RMD and Binary classiﬁer are better at OOD detection than baselines
Table 1 shows the
AUROCs for OOD detection for summarization and translation.
Overall, our proposed OOD score outperforms the baselines with high AUROCs.
The commonly used output metrics, perplexity, NLI,
COMET and Prism, have generally low AUROCs, suggesting they are not suited for OOD detection.
Interestingly, we noticed that the output OOD scores perform better for summarization, while the input OOD scores perform better for translation.
Note that all methods have small AUROC for law dataset, suggesting that none of the methods can detect it as OOD.
However we found law has the highest unigram overlap (48.8%) with the in-domain data (Table A.7).
This conﬁrms that law is actually not OOD data and explains why no method can detect it.
3.2 Using OOD scores for selective generation
Though completely abstaining from OOD inputs is a conservative option, it is more desirable to expand the use of models beyond strictly in-distribution examples, if output quality is sufﬁciently [Table] high.
In classiﬁcation, this has been framed as determining when to trust a classiﬁer, or selective prediction (Geifman & El-Yaniv, 2017; Lakshminarayanan et al., 2017; Tran et al., 2022). Here we seek to predict the generation quality given a potentially OOD example
[Table]
and abstain if the quality is low.
We call this selective generation .
To measure output quality, we use BLEURT (Pu et al., 2021) for translation, and human evaluation for summarization (details in Section A.2.1) [Table]
We ﬁrst observe that perplexity has diminishing capability in predicting quality on OOD data .
Since the models are trained using negative log-likelihood as the loss, perplexity is a good predictor of output quality for in-domain.
However, we found that perplexity is worse at predicting quality on shifted datasets.
Figure A.5 plots the correlation between perplexity and quality as a function of
OOD score.
As OOD score increases, a decreasing correlation is observed for both summarization and translation.
We further observed that our OOD score and perplexity are complementary in quality prediction .
As illustrated in Figure A.6, neither perplexity nor OOD score can perfectly separate good and bad examples, and the combination of the two can work much better.
We propose two simple methods to combine perplexity and OOD scores .
(1) A simple linear regression, (2) the sum of the percentile ranks (PR) of the scores, i.e. PR sum =
PR perplexity + PR OOD .
Table 2 shows the correlation coefﬁcient between the various single and combined scores and the quality metric with only in-domain and all examples from all datasets.
When all datasets are merged, the combined scores signiﬁcantly improve the correlation over perplexity by up to 12% (absolute) for summarization and 8% for translation, while the gains over the best external model-based (and much more expensive) baselines are 4% and 3% .
The two combination methods perform similarly.
See Tables A.2 and A.3 for an expanded table of scores.
To evaluate selective generation, we propose the Quality vs Abstention Curve (QA) : at a given abstention rate α , the highest α -fraction scoring examples are removed and the average quality of remaining examples is computed.
We want to maximize the quality of what is selectively generated and a better curve is one that tends to the upper-left which corresponds to removing bad examples earlier than good ones.
As shown in Figure 1 (a,c), the combined scores have the highest quality score at almost all abstention rates for both summarization and translation , and the linear regression and the PR sum perform similarly.
Interestingly, we see our simple combined score is even marginally better than COMET, which is a separate neural network model trained on annotated human evaluation data while our combined score does not use any human annotation for supervision.
Area under the QA curves are shown in Tables A.4 and A.6 for reference.
Figures 1 (b, d) are the corresponding survival curves showing how many examples per dataset are selected for generation as a function of abstention rate, based on the PR sum score.
OOD and worst-quality data are eliminated ﬁrst and the in-domain data are absteained last.
The order in which datasets are eliminated corresponds to the aggregate quality by dataset , which we report in Table A.1. Acknowledgements
We thank Jeremiah Liu, Dustin Tran, Sharat Chikkerur, Colin Cherry, George Foster, and the anonymous reviewers for helpful feedback and discussions.
[Figure]
A Appendix A.1 OOD Detection in Conditinoal Language Models A.1.1 Perplexity is ill-suited for OOD detection
In Figure A.1, we compare the distribution of perplexity of (a) a summarization model and (b) a translation model trained on in-domain dataset and evaluated on multiple OOD datasets, respectively.
For summarization, a model is trained on xsum and evaluated on other news datasets including cnn_dailymail and newsroom as near-OOD datasets, and forum ( forumsum ) and dialogue ( samsum and reddit_tifu ) datasets as far-OOD (see Section 3 for details).
The perplexity distributions overlap signiﬁcantly with each other even though the input documents are signiﬁcantly different.
Furthermore, perplexity assigns cnn_dailymail even lower scores than the in-domain xsum .
For translation, the model is trained on WMT15 dataset and evaluated on other WMT test splits (Bojar et al., 2015), OPUS (Aulamo & Tiedemann, 2019), and MTNT (Michel & Neubig, 2018).
The in-domain and OOD datasets perplexity densities overlap even more.
Overall, these results suggest that perplexity is not well suited for OOD detection.
[Figure]
for OOD detection due to signiﬁcant overlap between in-domain and OOD scores.
A.1.2 Our proposed OOD detector based on input and output embeddings
[Figure]
Binary classiﬁer for OOD detection Since we have implicitly deﬁned two classes, in-domain and background/general domain, another option is to train a binary classiﬁer to discriminate embeddings from the two classes.
We train a logistic regression model and use the un-normalized logit for the background as an OOD score.
The Input Binary logits
OOD score uses the input embeddings as features, whereas the Output Binary logits OOD score uses the decoded output embeddings as features.
A higher score suggests higher likelihood of OOD.
The preferred use of the logits over probability was also recommended by previous OOD studies for classiﬁcation problems (Hendrycks et al., 2019).
Though RMD is a generative-model based approach and the binary classiﬁer is a discriminative model, we show that RMD is a generalized version of binary logistic regression and can be reduced to a binary classiﬁcation model under certain conditions
Though RMD and Binary logits OOD scores both perform well at OOD detection, RMD OOD score is better at distinguishing near-OOD from far-OOD .
This can be seen in Figure A.3 where nearOOD datasets have scores distributed in between in-domain and far-OOD.
In the summarization task, near-OOD (news articles) datasets cnn_dailymail and newsroom have their RMD scores distributed in the middle of xsum and reddit _ tifu , forumsum and samsum .
In contrast, under the binary logits score, the near-OOD and far-OOD datasets have largely overlapping score distributions making it hard to distinguish between the two.
In practice, RMD OOD score may be better suited for selective generation where domain shifts are expected.
We explore this in more detail in Section 3.2.
[Figure]
The connection between RMD and Binary classiﬁer RMD is a generative model based approach which assumes the distributions of the two classes are Gaussian, while the binary classiﬁer is a discriminative model which learns the decision boundary between two classes.
Though they have different settings, under certain condition, the Gaussian generative model can be reduced to a binary classiﬁer.
To see the connection, let us assume the label y
=
if the sample is from in-domain, and y = 1 if the sample is from the general domain.
Let us also assume the two classes have balanced sample size without loss of generality p
( y = 1) =
p ( y = 0) .
Since the log-probability of log p ( y = 1 | z ) can be rewritten using the Bayes rule log p ( y = 1 | z ) = log p ( z | y = 1) + log p ( y = 1) − log p ( z ) , the logit (log odds) can be written as,
[Equation]
When Σ = Σ
y
=1 = Σ y =0 , the equation can be further simpliﬁed as
[Equation]
Therefore, when assuming the covariance matrices are identical for the two Gaussian distributions, the Gaussian generative model can be reduced to a binary classiﬁcation model.
However, our RMD does not assume the same covariance matrix in both distributions.
We estimate the covariance matrix individually for each class.
So our RMD is different from binary classiﬁer, and it has higher model capacity than the binary classiﬁer.
A.2 Using OOD scores for selective generation A.2.1 Quality metrics used for translation and summarization Measuring Translation quality
We use BLEURT (Pu et al., 2021) as the main metric to measure translation quality.
Previous work has demonstrated that neural metrics such as BLEURT are much better correlated with human evaluation, on both the system level and the sentence level (Freitag et al., 2021).
BLEURT scores range from 0 to 1, with higher scores indicating better translation quality.
Measuring Summarization quality
In general, it is unclear how to automatically measure the quality of summaries generated by a model on out-of-distribution examples (in this case, examples from different datasets).
The reason is summarization datasets have dataset-speciﬁc summary styles that may be difﬁcult to compare.
For example, xsum summaries are typically single-sentence whereas cnn_dailymail summaries consist of multiple sentences.
Thus we report ROUGE-1 score as an automatic measure but primarily use human evaluation to assess the quality.
Amazon Mechanical Turk workers were asked to evaluate summaries generated by the xsum model on a scale of 1-5 (bad-good) using 100 examples from xsum , cnn_dailymail , reddit_tifu , and samsum .
We collected 3 ratings per example and took the median to reduce inter-rater noise.
Speciﬁcally, a PEGASUS LARGE model ﬁne-tuned on xsum was run on a random sample of 100 examples from the test split of four datasets: xsum , cnn_dailymail , reddit_tifu , samsum .
Each example was rated for general summarization quality on a rating of 1-5 by 3 AMT workers using the template shown in Figure A.4.
Workers were required to be Masters located in the US with greater than 95% HIT
Approval Rate, with at least 1000 HITs approved and were paid $0.80 per rating.
The median rating was used to reduce noise.
[Table] [Table]
A.2.2 Perplexity has diminishing capability in predicting quality on OOD data
Since the models are trained using negative log-likelihood as the loss, perplexity (which is monotonically related) is a good predictor of output quality for in-domain data.
In fact, the Kendall rank correlation coefﬁcient τ between perplexity and human judged quality score is 0.256 (See Table 2) for in-domain xsum for summarization.
However, when including shifted datasets to test, we found that the perplexity score is worse at predicting quality on OOD data.
For example the Kendall’s τ decreases to 0.068 for OOD dataset samsum (see Table A.2).
We observed similar trend in translation, although less severe, as data shifted from in-domain to OOD, the Kendall’s τ between perplexity and
BLEURT decreases (see Table A.3).
Figure A.5 further shows the correlation between perplexity and the quality score (ROUGE-1, human rating, and BLEURT, respectively) as a function of OOD score.
It is clear to see the correlation decreasing as OOD score increases and the trend is consistent for both summarization and translation.
[Figure]
2.3 OOD score and perplexity are complementary for predicting output
[Figure]
A.2.4 Correlation between different scores and the quality metrics
[Table]
[Table] [Table]
[Figure]
[Table]
[Figure]
A.2.5 Selective generation and output quality prediction
[Figure] [Figure]
[Table] [Table]
[Table]
A.3 Investigation of the n-gram overlap between law dataset and in-domain datasets
[Table]
A.4 Visualization of OOD score on shifted dataset
We explore how individual parts of an input text contribute to the OOD score, which can help us visualize which parts of the text are OOD.
We deﬁne the OOD score of each sentence in the text using a leave-one-out strategy:
For any given sentence, we compute the OOD score of the article with and without that sentence in it.
The negative of the change in the OOD score after removing the sentence denotes the OOD score of that sentence.
Intuitively, if removing the sentence decreases the overall OOD score, that sentence is assigned a positive OOD score and vice-versa.
Figure A. illustrates an example where an article contains noise in the form of tweets with emojis
A.5 Related Work
OOD detection problem was ﬁrst proposed and studied in vision classiﬁcation problems (Hendrycks & Gimpel, 2016;
Liang et al., 2017; Lakshminarayanan et al., 2017; Lee et al., 2018; Hendrycks et al., 2018, 2019), and later in text classiﬁcation problems such as sentiment analysis (Hendrycks et al., 2020), natural language inference (Arora et al., 2021), intent prediction (Liu et al., 2020a; Tran et al., 2022), and topic prediction (Rawat et al., 2021).
The widely used OOD methods can be characterized roughly into two categories (1) softmax probability or logits-based scores (Hendrycks & Gimpel, 2016;
Liang et al., 2017; Hendrycks et al., 2019;
Liu et al., 2020b), (2) embedding-based methods that measure the distance to the training distribution in the embedding space (Lee et al., 2018; Ren et al., 2021; Sun et al., 2022). [Table]
OOD detection problem is relatively less studied in CLMs.
A few studies explored the OOD detection in semantic parsing (Lukovnikov et al., 2021; Lin et al., 2022), speech recognition (Malinin & Gales, 2020), and machine translation (Malinin et al., 2021; Xiao et al., 2020), but many of them focus on ensemble-based methods like Monte Carlo dropout or deep ensemble which sample multiple output sequences auto-regressively and use the averaged perplexity as the uncertainty score.
The ensembling method costs N times of the inference time, which is not feasible in practice.
In this work, we focus on developing scores that can be readily derived from the generative model itself, without much increase in computation. A.6 Comparison with KNN-based OOD score MD and RMD assume the embedding follows the parametric Gaussian distribution.
Here we compare them with the non-parametric OOD detection methods which do not assume a parametric distribution.
One of the non-parametric methods is KNN-based OOD score proposed in Sun et al.
There are two hyper-parameters in the KNN-based method, α and k .
α is the proportion of training data sampled for nearest neighbor calculation, and k refers to the k -th nearest neighbor.
We use the optimal k = 1000 and α = 100 as suggested by the paper.
We also normalize the embedding features since the paper showed the feature normalization is critical for good performance.
Table A.8 shows the AUROCs for OOD detection using the KNN-based method and comparing that with MD and
RMD methods.
As shown in the table, for the input OOD, KNN performs better than MD and RMD in cnn_dailymail , but worse than the two for other OOD datasets.
For the output OOD, KNN is worse than both MD and RMD.
It is possible that the optimal hyper-parmaters suggested by the paper may not be the optimal ones for our problem, and a ﬁne-grained hyper-parameter search could achieve better performance.
We leave this for future study.
A.7 Summarization examples with low/ high predicted quality scores
Besides the quantitative results, here we show a few real examples to better demonstrate how well our predicted quality score helps for selective generation.
Figure A.10, A.11, and A.12 show 3 examples in cnn_dailymail that have the highest
PR sum (perplexity, output RMD) scores that predict for low quality summaries.
Figure A.13, A.14, and A.15 show 3 examples in cnn_dailymail that have the lowest
PR sum (perplexity, output RMD) scores that predict for high quality summaries.
Document: A man trying to elude police jumped into a Missouri creek overnight wearing only his underwear – but his daring gambit did not pay off.
Responding ofﬁcers and ﬁreﬁghters followed the fugitive into the murky waters of Brush Creek in Kansas City and ﬁshed him out early Friday morning.
The 38-year-old suspect has been taken to an area hospital to be treated for injuries to his arm and leg.
He may face charges in connection to a hit-and-run crash.
Escape by water: A 38-year-old man stripped down to his skivvies and jumped into Brush Creek in Kansas City, Missouri, after being stopped by police.
Up Brush
Creek without a paddle: The suspect reached the middle of the creek and spent 10-15 minutes swimming back and forth.
According to a Kansas City Police Department’s arrest report, ofﬁcers were called to a gas station in the 4600 block of Prospect at around 2am after receiving complaints from neighbors about a car blasting loud music.
The report states that when police approached the car, a grey 2007 Inﬁnity, and asked to see the driver’s license, the man smiled, said, ‘I’m out!’
and took off from the scene.
The Inﬁnity promptly smashed into the north side of the Brush Creek bridge, after which the driver got out of the mangled car and jumped into the water.
Police say the 38-year-old suspect stripped down to his underwear and spent 10-15 minutes swimming in chest-deep water, with ofﬁcers waiting for him on north and south sides of the creek.
Surrounded: When ﬁreﬁghters tried to pull him out, he threatened them with a log.
Fish out of water: Police ofﬁcers armed with a BB gun went after the nighttime bather and apprehended him.
The bather was complaining of a broken leg, according to Fox4KC, so the Kansas City Fire Department’s water rescue crew were sent in to ﬁsh him out.
But the half-naked man in the water was not going to go quietly.
‘The suspect picked up a large log and started swinging it at the ﬁremen so they backed off as to not escalate the situation,’ the arrest report states.
That is when uniformed police ofﬁcers armed with a
BB gun followed the man into the creek, got him in a choke hold and pulled him out of the creek.
Police suspect the man may have been under the inﬂuence of drugs or alcohol.
Prelude: Before he jumped in the water, the 38-year-old driver ﬂed from police and smashed his 2007 Inﬁnity into a bridge.
Police suspect the man may have been under the inﬂuence of drugs or alcohol at the time.
As of Friday morning, the 38-year-old has not been formally charged with any crime.
Reference Summary: The 38-year-old suspect was questioned by Kansas City police after neighbors complained he was blasting music in his 2007 Inﬁnity.
Instead of handing over his ID, driver smiled, said ’I’m out!’
and took off.
After crashing into bridge, the man stripped down to his underwear and jumped into Brush Creek.
It took cops armed with a BB gun 15 minutes to ﬁsh out the fugitive.
Model Summary: All images are copyrighted.
[Figure]
[Figure]
PR
[Figure]
(perplexity,
[Figure]
RMD) ( ↓ means high quality):
[Figure]
Document: A crisp fan who gets through 42 bags in a week has discovered a skull-shaped deep-fried potato snack in one of his packets.
Barry Selby, 54, who lives with his dog in Poole, Dorset, was eating a bag of cheese and onion crisps when he made the bizarre discovery, which appears to be a proﬁle of a human skull.
The ﬂoor-ﬁtter has decided to keep the two inches tall by two-and-a-half inches wide snack as he believes it is far more impressive than other oddly-shaped examples he has seen on the internet.
Scroll down for video.
Spooky ﬁnd: Barry Selby was eating a bag of Tesco cheese and onion crisps when he found the ’skull’ snack.
Mr Selby said: ’I was shocked when I found it.
I was just eating a bag of cheese and onion crisps from Tesco and when I pulled it out it did take me back a bit. ’
I thought it was worth keeping as I don’t think I will ever ﬁnd one like it again.
It must have been a very weird-shaped potato. ’
It’s about two inches tall and two-and-a-half inches wide and it’s in perfect detail, it even has an eye socket. ’
I sometimes give my dog, Max, crisps in a bowl, so it’s lucky he didn’t have this packet
or I wouldn’t have found it.
Weird snack: Mr Selby has decided to keep the unusual ﬁnd, which appears to show a jaw, nose and eye.
Comparison: The 54-year-old said he was ’shocked’ to make the discovery, although it is not his ﬁrst.
In the 1990s he came across a 3D heart-shaped crisp, which he kept until it broke.
And it’s not the ﬁrst odd-shaped snack he has come across - in the 1990s he found a crisp shaped like a 3D heart, which he kept for several years until it broke.
But he says this ﬁnd was different: ’This one was a big one.
I just thought "wow" and wanted to share it. ’
I’ve been keeping it on top of my computer in the front room, but it should be in a protective box really. ’
I’m going to keep it forever, it’s just so spooky.
I looked on the internet for other funny-shaped crisps but this is a one-off.’
Reference Summary: Barry Selby from Dorset was eating bag of Tesco cheese and onion crisps.
The 54-year-old discovered a snack shaped like proﬁle of the human skull.
He said he was ’shocked’ with the ﬁnd and has decided to ’keep it forever’
It’s not his ﬁrst weird food ﬁnd - he once discovered a heart-shaped crisp.
Model Summary: All images are copyrighted.
Human rating
[Figure]
( ↑ means high quality)
[Figure]
[Figure]
PR
[Figure]
(perplexity, output RMD)
[Figure]
means high quality)
[Figure]
Document: Last week she was barely showing – but Demelza Poldark is now the proud mother to the show’s latest addition.
Within ten minutes of tomorrow night’s episode, fans will see Aidan Turner’s dashing Ross Poldark gaze lovingly at his new baby daughter.
As Sunday night’s latest heartthrob, women across the country have voiced their longing to settle down with the brooding Cornish gentleman – but unfortunately it seems as if his heart is well and truly off the market.
Scroll down for video.
Last week she was barely showing – but Demelza Poldark is now the proud mother to the show’s latest addition.
He may have married his red-headed kitchen maid out of duty, but as he tells her that she makes him a better man, audiences can have little doubt about his feelings.
What is rather less convincing, however, is the timeline of the pregnancy.
With the climax of the previous episode being the announcement of the pregnancy, it is quite a jump to the start of tomorrow’s instalment where Demelza, played by Eleanor Tomlinson, talks about being eight months pregnant.
Just minutes after – once again without any nod to the passing of time – she is giving birth, with the last month of her pregnancy passing in less than the blink of an eye.
With the climax of the previous episode being the announcement of the pregnancy, it is quite a jump to the start of tomorrow’s instalment where Demelza, played by Eleanor Tomlinson, talks about being eight months pregnant.
As Sunday night’s latest heartthrob, women across the country have voiced their longing to settle down with Poldark – but unfortunately it seems as if his heart is well and truly off the market.
Their fast relationship didn’t go unnoticed by fans.
One posted on Twitter: ‘If you are pregnant in Poldark times expect to have it in the next 10 minutes’ It is reminiscent of the show’s previous pregnancy that saw
Elizabeth, another contender for Ross’s affection, go to full term in the gap between two episodes.
This didn’t go unnoticed by fans, who posted on Twitter: ‘Poldark is rather good, would watch the next one now.
Though if you are pregnant in Poldark times expect to have it in the next 10 minutes.’
[Table]
[Table]
gives birth to baby on Sunday’s episode.
Only announced she was pregnant with Poldark’s baby last week.
Model Summary: It’s all change in the world of Poldark.
Human rating score ( ↑ means high quality): 0.
PR sum (perplexity, output RMD) (
[Figure]
means high quality):
[Figure]
Document: Rangers boss Stuart McCall says he is already working on a dossier of signing targets for next season - even though he may not be around to parade them.
The interim Ibrox manager still does not know if he will be in charge beyond the current campaign after being lured back to his old club to kick-start their faltering promotion bid.
So far, everything is going to plan with Gers second in the Scottish Championship table and destined for a semi-ﬁnal play-off slot.
Stuart McCall says he is already looking at transfer targets for next season, though he may not be at Rangers.
But with 12 players out of contract, McCall knows the Light Blues will need to strengthen if they have any chance of keeping pace with rivals
Celtic next season - if they go up - and is already piecing together a wish list of potential new arrivals.
He said: ’I’ve been speaking to a lot of agents and putting things in place for if and when...
Even if I’m not here, if I’m getting players put to me who would like to come to Rangers regardless of the manager, then we build a little portfolio of positions that we will be needing next year. ’
It’s not a case of us standing still and then thinking come June 1, ’Oh we need to get into action’. ’
No, there are a lot of agents who come to us and we build a little dossier of players that as a staff, we think will be good for next season, regardless of what league we are in. ’
It would be slightly naive [if we were not doing that].
If I’m in charge or not, I still want the club to do well and I will put my view across to the board on who I think should be coming into the club and who should be here.’
McCall is compiling a dossier on targets as he looks to put the club in the best possible position.
Rangers have operated a haphazard transfer policy since re-emerging from the embers of liquidation.
The club’s team of scouts were jettisoned under the disastrous Craig Whyte regime and former boss Ally McCoist was largely forced to turn to a list of former Ibrox servants he had personal knowledge of when trying to bolster his squad.
But McCall revealed the club’s new board are now starting the process of re-establishing their spying network - albeit on a smaller level than before.
’I think there has been discussions behind the scenes with different people,’ said the former Motherwell boss. ’
I don’t think we are at the stage where we were 10 or 15 years ago where we were aiming to get into the Champions League and bringing players in for three and four million yet. ’
I don’t think Rangers will be at the stage yet next year where we need international scouts everywhere.
Rangers have expanded their scouting network after a haphazard system over the past few years. ’
But certainly a scouting network needs to be put in place. ’
Having said that, I spoke to Craig Levein at Hearts and they do a lot of their scouting with [online service] Wyscout.
When I brought Henrik Ojamaa in at Motherwell, that was after
I’d seen a clip of him on YouTube.
I sold him for £350,000 after signing him for nothing.
That was great.
’So you can still do your own background work.
Personally I would always like to see the player myself.
I’ve only ever signed one player without watching him ﬁrst and slightly regretted it. ’
So yeah we need a scouting network but at this moment where Rangers are, not to the extent where we have scouts all over
Europe.’
McCall admitted he still does not know if he will rejoin Gordon Strachan’s Scotland staff for the June 13 Euro 2016 qualiﬁer with Ireland in Dublin.
And he also confessed to uncertainties ahead of
Saturday’s match with Falkirk.
McCall’s side are still in line for promotion, sitting in the play-off positions in the Scottish Championship.
Peter Houston’s Bairns - ﬁve points behind fourth-placed Queen of the
South with two games to play - need an unlikely series of results to make the play-offs but McCall says that raises more questions than answers.
He said: ’Housty is a wily old fox who has done terriﬁcally well in his career
so I don’t know what to expect. ’
It will take a difﬁcult set of results for them to get into the play-offs so I don’t know if they will come here and think the pressure is off and play care free. ’
They don’t lose many goals so we may have to be patient through the 90 minutes.
We have had a couple of decent results against them but they have capable players and we will need to be at our best.’
Reference Summary:
Rangers are currently second in the Scottish Championship.
Stuart McCall’s side are in pole position to go up via the play-offs.
But McCall is still not certain of his future at the club next season.
Rangers boss says he is still trying to build the squad for next year.
Rangers have begun to expand their scouting after several poor years.
Model Summary: Stuart McCall says he is already looking at transfer targets for next season, though he may not be at Rangers.
Human rating score ( ↑ means high quality):
[Figure]
PR
[Figure]
[Figure]
[Figure]
Document: An Alberta student who’d accidentally left his headlights on all day was greeted by what may have been the world’s friendliest note from a stranger when he returned to his car.
But Derek Murray, a
University of Alberta law student, found more than just the note that cold November day in Edmonton–he also found an extension cord and battery charger left by the stranger to bring his dead Acura back to life.
Now that Murray’s life-afﬁrming tale has now gone viral, he says ’It just shows you how such a pure act of kindness from one person can just spread through everyone and help make everyone’s day a little brighter.’
Good Samaritan:
A friendly stranger left this unbelievably friendly letter to Alberta law student Derek
Murray in order to help him get his car started after he left the headlights on all day.
At ﬁrst, though, he assumed the letter was from an angry fellow motorist, he told the National Post. ’
When I ﬁrst saw the note, I was expecting it to be an angry letter from someone telling me not to park there.
Instead, I got someone just totally brightening my day.
My day could have been ruined but, because of this guy, it was the highlight of my day.’
The note reads, in part:.
I noticed you left your lights on.
The battery will probably not have enough charge to start your vehicle.
I left a blue extension cord on the fence and a battery charger beside the fence in the cardboard box.
If you know how to hook it up, use it to start your car.
What followed was a detailed explanation of how to use the equipment. ’
Sure enough,’ Derek recalled to the National Post, ’I looked over at the house my car was parked beside, and there was a blue extension cord plugged into an outlet behind the guy’s house with a battery charger right there beside it.’
Derek was able to get his car started, but when he rang the good Samaritan’s doorbell, there was no answer.
So, Derek left his own note as a thank you for the kind gesture.
He later snapped a photo of the stranger’s friendly note to post to Facebook, where it has now gone viral.
The note has been viewed millions of times and even Edmonton Mayor Don Iveson retweeted the photo.
Derek snapped a photo of the note for Facebook
and it has since gone viral.
e ’It just shows you how such a pure act of kindness from one person can just spread through everyone and help make everyone’s day a little brighter,’ Derek said.
Reference Summary: Derek Murray, a University of Alberta law student, could have had his day ruined by the mistake by a stranger’s kindness brightened it up.
Murray posted his story and the note online and the random act of kindness has now gone viral.
Model Summary: A Canadian student who accidentally left his headlights on all day was greeted by what may have been the world’s friendliest note from a stranger when he returned to his car.
Human rating score ( ↑ means high quality): 0.
PR sum (perplexity, output RMD) ( ↓ means high quality):
[Table]
that predict for summary
Document: Bayern Munich had to make do without FOUR important ﬁrst-team stars as Pep Guardiola’s side attempted to overturn a 3-1 deﬁcit against Porto on Tuesday night.
Injured quartet Franck Ribery,
Mehdi Benatia, David Alaba and Arjen Robben were forced to watch on from the sidelines as the German giants bid to reach the Champions League semi-ﬁnals.
However, the absence of Robben and Co appeared to make no difference as Bayern raced into a 5-0 lead at half-time before claiming a 6-1 victory to win the tie 7-4 on aggregate.
Injured trio Franck Ribery, Mehdi Benatia and David Alaba chat ahead of Bayern’s clash with Porto.
Injured Ribery acknowledges a steward before taking a seat at the Allianz Arena on
Tuesday night.
Ribery looks on as former Roma defender Benatia chats with the France international in the dugout.
While Ribery, Benatia and Alaba chatted in the home dugout ahead of kick-off, Holland international Arjen Robben was in front of the mic doing some punditry alongside Bayern goalkeeping legend Oliver Kahn.
Ribery missed the game after failing to recover from a recent ankle injury while former Roma defender Benatia faces another two weeks out with a groin problem.
Robben was unavailable for the encounter with an abdominal injury.
David Alaba, meanwhile, is set for a month on the sidelines having partially ruptured knee ligaments playing for Austria at the start of April.
Bayern had just 14 ﬁt players to choose from against Porto in the ﬁrst leg but tore the Portuguese giants apart at the Allianz
Arena to progress.
Holland international Arjen Robben was pictured doing punditry alongside Bayern legend Oliver Kahn (right)
Bayern Munich wideman Robben was unavailable for the Champions League clash with an abdominal injury.
Reference Summary: Bayern Munich beat Porto 6-1 at the Allianz Arena on Tuesday night.
German giants were without Franck Ribery, David Alaba and Mehdi Benatia.
Arjen Robben was also sidelined and did some punditry for the tie.
Model Summary: Arjen Robben, Mehdi Benatia, Franck Ribery and David Alaba all missed Bayern
Munich’s Champions League quarter-ﬁnal
[Figure]
rating score (
[Figure]
means high quality)
[Figure]
perplexity
[Figure]
output RMD)
[Figure]
high
[Figure]
[Figure]