Jack-Ryder at S em E val-2023 Task 5: Zero-Shot Clickbait Spoiling by Rephrasing Titles as Questions (2023)

Abstract: In this paper, we describe our approach to the clickbait spoiling task of SemEval 2023.The core idea behind our system is to leverage pre-trained models capable of Question Answering (QA) to extract the spoiler from article texts based on the clickbait title without any task-specific training. Since oftentimes, these titles are not phrased as questions, we automatically rephrase the clickbait titles as questions in order to better suit the pretraining task of the QA-capable models. Also, to fit as much relevant context into the model’s limited input size as possible, we propose to reorder the sentences by their relevance using a semantic similarity model. Finally, we evaluate QA as well as text generation models (via prompting) to extract the spoiler from the text. Based on the validation data, our final model selects each of these components depending on the spoiler type and achieves satisfactory zero-shot results. The ideas described in this paper can easily be applied in fine-tuning settings.

## introduction the objective of the clickbait spoiling task @xcite is to find/extract the spoiler from a linked article that satisfies the reader's curiosity that was induced by a clickbait title (e.g. "you won't believe how well this semeval clickbait spoiling system performs!"). the challenge differentiates between three types of spoilers: phrase (the spoiler is a short phrase), passage (a sentence or multiple sentences that span from a beginning position to an end position without any gap), and multi (the spoiler consists of multiple disjoint phrases/passages). in this paper, we describe our system for this challenge that is based only on pre-trained models and no fine-tuning. we model spoiler extraction as a question answering (qa) task and explore four different pre-trained architectures: two extractive qa using bert models and two prompt-based text generation models (with two prompts each). in order to make the clickbait title more similar to the questions the qa models have been trained on, we propose to automatically rephrase them into a question form. to ensure that the model receives the most relevant article text passages in its limited context, we propose to filter and sort the article text using a pre-trained semantic similarity model. we evaluate our pipeline on the validation dataset and combine the best components for each spoiler type as our final model. we find that rephrasing the clickbait title usually improves performance, while relevance-based context ordering does not. our final model makes use of clickbait title rephrasing for phrase and multi-spoiler types, a deberta qa model for phrase and passage-type spoilers, and a flan-t5 model for multi-type spoilers. the results on the test set show satisfactory results, considering that our system is not fine-tuned in any way. our code is publicly available foot_0 . ## background research on clickbait texts can be divided into mainly three categories: clickbait detection, type classification, and spoiling. clickbait detection tries to predict whether a news article's headline is a clickbait title. for this, several feature-based machine learning models have been developed @xcite @xcite . clickbait type classification and spoiling are the tasks of this challenge, which try to detect the type of spoiler (phrase, passage, multi) and extract the spoiler from the text, respectively. for clickbait type classification, transformer-based models have shown the best performance @xcite . the information about the type of spoiler can potentially be used in the spoiler extraction model. clickbait spoiling is mainly done using fine-tuned extractive and abstractive question answering (qa) models @xcite clickbait title clickbait title rephrasing pretrained qa-capable model question answering or text generation (with prompting) relevance-based context ordering article text spoiler figure 1 : overview of our system. it consists of mainly three components that are enabled based on the spoiler type. given a clickbait title and its corresponding article text, the title is potentially rephrased into a question. to ensure that the article text fits into the context size of the following qa-capable model, we optionally use a sentencebert model to sort the text's sentences by relevance. the question and text are used in a pre-trained qa-capable model, which is either a qa model or a text generation model that uses prompting to extract the spoiler. @xcite , where extractive models are generally more suitable @xcite . zero-shot spoiling using pre-trained qa models is inferior to fine-tuned models @xcite . all of the available approaches use the clickbait title without modification as the "question" input, which might not be beneficial for the performance of the model, since the clickbait title of news articles is oftentimes not phrased as a question (e.g. "you won't believe what happened after the event!"). in this paper, we show for the spoiling task that rephrasing the clickbait title as a question usually improves the performance of zero-shot qa models. while this still not achieves comparable performance to fine-tuned models, exploring easyto-implement steps to improve zero-shot performance can easily be adapted to fine-tuned settings. for the task, a training and validation set are provided, with each example consisting of a clickbait title (posttext; the text that is posted on social media), article text (targettitle and targetparagraphs), the extracted spoiler (spoiler; not available during testing) with its positions (spoilerpositions; not available during testing), as well as the type of spoiler (tags). additional meta-information is provided but not used by our system. all submissions are submitted to and evaluated on tira @xcite . ## system overview our system is shown in figure 1 . it only leverages pre-trained models (question answering, text generation, and semantic similarity models) to extract the spoiler from a given text without the need for any training. while it is certainly beneficial to employ fine-tuning as done by previous work, we experiment with other ideas that have not been proposed in the clickbait spoiling literature, leading to new insights into what works and what does not for this kind of task. the elimination of training enables faster model development and fast updates to the model pipeline when a better component is available. note that our system can also be used in fine-tuning settings without any modifications, so our proposed additions to the pipeline could be beneficial to fine-tuned approaches. our pipeline consists of mainly three components that are introduced based on the following key ideas: 1. qa models are trained on questions, but clickbait titles often are not questions. thus, we use a pre-trained model to rephrase clickbait titles into questions. 2. the article texts are oftentimes too long for the model input size. therefore, we order each article's sentences by relevance using semantic similarity to the title, selecting the most important sentences as model input. 3. recently, zero-shot capabilities of large language models are getting massive research interest, but have not been investigated for clickbait spoiling. therefore, we evaluate text generation-based models with different prompt templates in addition to qa models, calling them collectively "qa-capable models". we now go through each of these three components (clickbait title rephrasing, relevance-based context ordering, and pre-trained qa-capable model) and describe how they work. we later select the model to use and whether to use clickbait title rephrasing and relevance-based context ordering based on the validation results for each spoiler type (phrase, passage, multi). ## experiments and results we evaluate our proposed system on the validation data of the challenge and give the bleu-4 score for all and spoiler-type specific examples in here, we observe multiple findings that we use to build the final model for our submission. ## conclusion in this paper, we have presented our approach to the clickbait spoiling challenge. we have explored multiple strategies to improve the performance of pre-trained models without any fine-tuning. the results are not as good as fine-tuned models (see other competitors and the challenge baseline), but we have found that clickbait title rephrasing usually improves performance, which can potentially be applied to fine-tuned models as well. one strategy that has not worked was the relevance-based context ordering, which mostly decreases performance. we suspect that our measure of relevance is not reliable for question and answer pairs. exploring more reliable and effective measures might be interesting. another possible focus for future work might be to experiment with more prompts for text generation models. we have shown that different prompts can have a severe impact on the model's performance, so more careful prompt engineering might lead to similar performance as for the qa models and can then get fine-tuned. also, using larger and thus more powerful text generation models such as the hosted gpt-3 foot_3 @xcite or the open source opt-175b @xcite are options that showed promising results in preliminary experiments, but were not further explored due to monetary and computational limitations.