Diane Simmons at S em E val-2023 Task 5: Is it possible to make good clickbait spoilers using a Zero-Shot approach? Check it out! (2023)

Abstract: In this paper, we present a possible solution to the SemEval23 shared task of generating spoilers for clickbait headlines. Using a Zero-Shot approach with two different Transformer architectures, BLOOM and RoBERTa, we generate three different types of spoilers: phrase, passage and multi. We found, RoBERTa pretrained for Question-Answering to perform better than BLOOM for causal language modelling, however both architectures proved promising for future attempts at such tasks.

## introduction most of the companies that want to advertise themselves use the internet as means of contact to the customers. the reputation of such companies is a strong signal of information quality, but social information encoded in metrics like the number of likes or shares on a post act as an alternative source of credibility @xcite . in their work, @xcite call this environment clickbait media. this content, whether it is produced by mass media, large companies or single individuals, is more likely to be shared if it evokes high-arousal emotions @xcite . in this context, a clickbait is some internet content whose main purpose is to encourage users to follow or click on a link to a web page. in a study conducted by @xcite , the authors observed that stories with clickbait headlines were consistently judged to be of lower quality and lower credibility, but these differences lacked of statistical power in the majority of the cases. clickbait creates excitement in us, with devices such as cataphora, a type of anaphora, where the speaker mentions a general expression that will be further specified later in the utterance. an example of this is the following: "if you want some, the number of clicks in a website is a common metric of user engagement @xcite . in the information retrieval community it was shown that a user sees a document almost always after clicking it @xcite . therefore, we can expect the number of clicks in a website to be associated with a higher success. these clicks are commonly associated with economical revenue, which in some cases can be the economical support of many websites or platforms. 2 in this paper we present our approach for the shared task on clickbait spoiling @xcite . 3 following current trends in natural language processing (nlp), we approached the problem of clickbait spoiling as a zero-shot question answering or text-to-text generation problem, using some of the most popular transformer-based pretrained models available in the huggingface platform @xcite . 4 we tested our model in the tira platform @xcite , provided by the organizers, and the code of our approach is available on github. 5 we argue that if a computer can spoil a clickbait, we can interpret that it (at least, partially) understood the mentioned article, and it was able to produce an utterance that responds to the question in the headline. then, this work can be seen as a step towards full natural language understanding. this article is structured as follows: first we in-troduce some background information where we present different models that were initialy tested. later, in the system overview section we present the actual models that we decided to use for submission. in a later section called experimental setup we further describe some relevant details about the models. after that, we present and briefly discuss our results and we conclude the paper with some possible future directions. ## background the task at hand is to spoil clickbait headlines from a wide range of corpora in english language. this can be approached as a text-to-text generation problem or as a question-answering problem. ## system overview the final models used were all transformers: roberta pretrained on the squad 2.0 dataset for question-answering foot_8 @xcite and two bloom causal language models, one with 7.1 billion parameters and the full sized model with 179 billion parameters (bigscience, 2022). all models were set up on the cloud computing service, ucloud (sdu, 2023), which utilises intel xeon gold 6130 cpus. the predictions from the roberta and 7.1b bloom model were executed on this hardware, while the 179b bloom model was used through the huggingface inference api @xcite as the model would be too computationally heavy for our systems. all models were fetched from the transformers library for python @xcite ). foot_9 ## experimental setup for our attempt at generating clickbait spoilers, we focused on zero-shot learning and prompt engineering to see how they would compare. for the roberta model, the default parameters were used in a pipeline, using the post text as a question and the target paragraphs as contexts. the model would then produce one answer per question. for the bloom models, various versions of prompts were tested and the final prompt engineering was used for both sizes of bloom. the fields used in the prompts were the target paragraphs, target title, description, tags and post text. the prompts consisted of up to five elements in the following order: each element is separated by two line breaks and if an entry did not contain an element, e.g. tags, it was left out. we below describe the steps for creating the prompts. firstly, the target paragraphs were joined and limited in length if too long, taking the first 2500 characters and the last 1000. the reason for limiting the article were mainly due to the api not accepting too long prompts. the first and last portions of the article were then kept, as we expect that the spoiler was most likely to be in the beginning or ending paragraphs. secondly, if the post text and target title are identical, do not include the target title, otherwise, add "article title: (target title)" to the beginning of prompt. then, add the article body as "article: (target paragraphs)" and afterwards add "tags: (tags)" separated by a space and a comma. then, if the description is not consisting of the first few paragraphs of the article body, include "description: (description)". finally, if the post text ends with a question mark, end the prompt with "question: (post text) answer:", e.g. "question: how safe is your dna?". otherwise, if the post text includes the words "what" or "why", add "(tags):", the reasoning being to add headlines such as "why you shouldn't scare your cat with a cucumber:" as a statement for bloom to fill out. finally, if none of the above cases for the clickbait headline were true, add "statement to be answered: (post text) answer:". similar reasoning as previously, but attempting to format less versatile headlines, e.g. "statement to be answered: back to jail". the bloom models used maximum 20 new tokens for generation and used greedy search for decoding, i.e. selecting the next word with the highest probability. the evaluation of the models while developing were mainly qualitative comparison between generated spoilers and target spoilers, both human spoilers and extracted spoilers, while generating the validation test. for the final prompt engineering, the included evaluation script was used for maximizing the bleu scores. the training data went unused for the final models. ## results if we look at the results, we can say that our model has its own flaws, but given the fact that it is a zero-shot approach, we believe that it works reasonably well. results are provided in the main conclusion that one can make is that the roberta model is the best one in almost all metrics. the bloom based models seem to work fairly similarly, making it hard to compare their performances. please find in tables 2 and 3, two examples with their respective gold spoilers and the outputs produced by our models. we could tentatively say from these two examples, that the roberta model is very precise in making short and concise predictions. the bloom models seem to have a low precision, but high accuracy, meaning that it does recall the right answer in its prediction, but it also include a substantial amount of information that is not relevant at all. future work could explore ways for reducing unnecessary information from the responses. ## conclusion in this paper we presented our approach for spoiling clickbaits. the approach does not make use of any task specific training as we employed a zero-shot approach. we tested three models, two bloom-based models and one roberta model, and results show that the roberta model works the best. after analyzing the outcomes of the models, we can say that the roberta model guesses rather well when the result is a named entity and that the bloom-based models are able to spot the result but they include quite a lot of irrelevant information. there are many possible future directions to continue the work presented here. for the prompt engineering many assumptions were made. we wonder whether it is a good idea to treat the headlines as questions or statements the way we did, or whether the headline should be presented in a different way. it would also be worthwhile to investigate few-shot learning by including one or more example articles with answers in the prompts. as for the greedy decoding used in bloom, other approaches could be used. the decoding method, beam search may be a better solution for the optimal clickbait spoilers.