Sabrina Spellman at S em E val-2023 Task 5: Discover the Shocking Truth Behind this Composite Approach to Clickbait Spoiling! (2023)

Abstract: This paper describes an approach to automat- ically close the knowledge gap of Clickbait- Posts via a transformer model trained for Question-Answering, augmented by a task- specific post-processing step. This was part of the SemEval 2023 Clickbait shared task (Frbe et al., 2023a) - specifically task 2. We devised strategies to improve the existing model to fit the task better, e.g. with different special mod- els and a post-processor tailored to different inherent challenges of the task. Furthermore, we explored the possibility of expanding the original training data by using strategies from Heuristic Labeling and Semi-Supervised Learn- ing. With those adjustments, we were able to improve the baseline by 9.8 percentage points to a BLEU-4 score of 48.0%.

## introduction this project was part of the semeval 2023 clickbait shared task @xcite . specifically, we focused on the task of generating spoilers for clickbait posts. we viewed this task as an extractive question-answering problem, since a knowledge gap (or question) posed by a headline shall be closed or answered by a passage from an article. based on this assessment, a transformer-based approach has been chosen since these approaches tend to perform well for question-answering problems @xcite . to improve the performance of our models, additional data for use in the training process has been generated. in addition, multiple models have been trained using different approaches. another challenge is that some of the examples require multiple spoilers to be generated as an answer. to address this issue, a post-processing approach to generate multiple relevant answers that are not too similar to each other has been developed. furthermore, a specific pattern could be identified and used to develop a heuristic in the pre-and post-processing step to find solutions to these examples independent of the model. ## background the task we attempted to solve was as follows: given a post text that introduces a so-called knowledge gap that piques the reader's interest in reading the accompanying article, our objective was to identify the passages within this article that best addresses this knowledge gap. ## system overview for this task, a typical bert-architecture consisting of a tokenization-and an encoding step was extended by a task-specific post-processing step. as mentioned in previous sections, we regard this task as a kind of question-answering problem. consequently, we interpret the post-text as a kind of question with regard to the article due to the knowledge-gap it contains. therefore, we fine-tuned a pre-trained transformer for questionanswering. in the following, we will describe how those models were trained and then how the mentioned post-processing step was designed. ## experimental setup the training for all our models was done using one a100 gpu. multiple models were trained for each 0% 10% 20% 30% 40% 50% bleu-4 score + enumeration spoiler + use tags + our base postprocessing transformer baseline naive baseline model versions 46.2% 44.5% 39.3% 38.2% 2.1% to evaluate the performance of a model, we used the bleu-4 score @xcite which is a similarity measure for strings using cumulative n-grams up to @xmath0. the models were trained with different combinations of input data. furthermore, for some models, hyper-parameter tuning was performed (see for running the spoiler prediction, a trained model is required. our models were packaged together with the code into docker containers and then submitted to tira @xcite . the following evaluation in section 5 is based on the validation clickbait dataset containing 800 samples. ## results two baselines are available for comparison @xcite ): a naive one that returns the title of the article as a spoiler and a transformer-based one that, like ours, uses the question-answering approach. figure 1 shows the results of these baselines as well as the evaluation of our post-processing using the transformer baseline model. the transformer baseline (38.2%) significantly outperforms the naive one (2.1%). overall, our post-processing improves the transformer baseline to a bleu-4 score of 46.2%. most of the improvement is caused by the use of the spoiler tags (text length limit and multispoiler iterations) and the rule-based enum-spoiler generation. baseline transformer model best universal model best specialized models for phrase and passage, bert-model for multi). the results for different subsets of the validation dataset are shown in figure 2 . both the universal model and the specialized models improve the baseline with our postprocessing to 47.2% and 48.0% respectively. the advantage of using specialized models can be clearly seen for passage and multi-spoilers. however, in the case of phrase-spoilers, our specialized model could not improve the universal one and is even worse than the baseline. the enumeration spoiler generation achieves a bleu-4 score of 69.8% on the relevant 24 samples. compared to the otherwise low scores for multi-spoilers, this represents a remarkable improvement. an interesting property related to the distribution of the bleu-4 score across each sample of the validation dataset is shown in figure 3 . our model finds (almost) perfect spoiler texts in about one third of the samples. on the other hand, another third performs really badly. the remaining samples are almost evenly spread between 10% and 90% bleu-4 score. further results based on the test dataset can be found in appendix a.5. ## conclusion with all the improvements applied to our baseline, we were able to increase the bleu-4 score for our predictions by 9.8 percentage points from 38.2% to 48.0%. using specialized models for different spoiler types 0 100 200 300 occurrences 0 1 2 3 4 5 6 7 8 9 10 70% -80% 80% -90% 90% -100% bleu-4 score 286 29 25 55 28 36 26 22 26 267 provides the opportunity to configure the training process for each model individually. the presented post-processing approach leverages model outputs to improve on these results further. even on the transformer baseline, this approach shows a solid improvement of 8 percentage points in the bleu-4 score. adding in the enumeration spoiler generation has proven to provide a considerable improvement in the results. overall, the use of transformer-based language models show solid results for this task. combining this approach with additional heuristics for specific cases and creating a composite approach shows great promise. a number of opportunities exist to improve this approach even further. for example, at the moment the enumeration spoiler generation as part of the post processing is only performed on samples tagged as enumeration type questions. refining the heuristic and applying it to all samples could yield further improvements (see @xcite ). an additional possibility to enhance the performance of the model is improving the dataset further, e.g. by using a better ner model that can differentiate more classes than the one we used. by checking for classes like actors, tips, and twitter accounts we could find more cscp candidates and create a larger, as well as more diverse cscp dataset, which represents the official cscps better.