S em E val-2023 Task 5: Clickbait Spoiling (2023)

Abstract: In this overview paper, we report on the second PAN~Clickbait Challenge hosted as Task~5 at SemEval~2023. The challenge’s focus is to better support social media users by automatically generating short spoilers that close the curiosity gap induced by a clickbait post. We organized two subtasks: (1) spoiler type classification to assess what kind of spoiler a clickbait post warrants (e.g., a phrase), and (2) spoiler generation to generate an actual spoiler for a clickbait post.

## introduction clickbait posts link to web pages and advertise their content by arousing curiosity instead of providing informative headlines or summaries. the pan clickbait challenge at semeval 2023 aims to develop techniques for automatic clickbait spoiling: generating a short text that can close the curiosity gap as exemplified in figure 1 . despite manual clickbait spoiling being popular on social networks, 1 the problem had not received much attention by the research community. we thus organized a respective challenge based on our previously published dataset and baseline approaches @xcite . with approaches submitted by 30 teams, the challenge had a very good participation. to improve reproducibility and replicability, we asked the teams to submit their software via the tira platform @xcite -already used in several shared task with an archive of more than 500 research prototypes. still, for our clickbait challenge, we also allowed result submissions, but a majority of the teams (19 of the 30) actually submitted software in the form of docker images. the task of clickbait spoiling, as exemplified in figure 1 , specifically aims at completing the "life cycle" of clickbait posts. the creation @xcite and detection @xcite of clickbait posts are already operationalized, so we 1 saved you a click has 1.9 million members on reddit. ## lab overview and statistics for the clickbait spoiling task, 83 teams from 24 countries registered. the majority of registrations came from the united states (15 registered teams), followed by germany and india (13 teams each), bangladesh (8 teams), romania (6 teams), china (4 teams), poland (3 teams), denmark, israel, mexico, russia (2 teams each), bulgaria, canada, colombia, france, japan, mexico, morocco, netherlands, norway, poland, spain, thailand, turkey (1 team each). overall, 26 of the 83 teams were undergrad students supervised by a senior researcher-most undergrad teams came from germany (8)and india (7). of the 83 registered teams, 30 actively participated in the task, out of which 23 teams submitted notebook papers describing their approaches. we used tira @xcite as our submission platform through which participants could either make software submissions or result submissions. software submissions have advantages over result submissions in terms of reproducibility (i.e., software can be rerun on different data) but also allow for blinded experiments 3 (i.e., ensuring that the test set is kept private). while we allowed result submissions, we strongly encouraged software submissions and made the test data only available upon request. overall, 10 active teams submitted results only, 1 team submitted results and software, while a majority of 19 teams submitted their software and never had access to the test data. to submit a software, a team implemented their approach in a docker image that they then uploaded to their dedicated docker registry in tira. software submissions in tira are immutable, and after uploading the image, the teams specified the to-be-executed command-the same docker image can thus be used for multiple software submissions (e.g., by changing some parameters). a team could upload as many docker images or software submissions as they liked; only they and tira had access to their dedicated docker image registry (i.e., the images were not public while the shared task was ongoing). overall, we received 202 software submissions from 86 unique docker images (the smallest image has a compressed size of 504 mb, the largest image has 45 gb). the teams could execute their software on the validation and the test data. on the validation data, the results, the evaluation scores, and the standard 3 wikipedia.org/wiki/blinded_experiment and error output were directly visible to be able to verify that a submitted software works as expected. on the test data, the evaluation scores and the outputs were blinded; the only available feedback was whether the software produced a valid output or not. to ensure that the test data is not leaked and to improve the overall reproducibility, tira executes software in a sandbox by removing the internet connection. this, for instance, helps to ensure that a software is fully installed in the docker image which eases rerunning a software at some later point in time (all libraries and models must be contained in an image to be able to execute the software without internet connection). we assisted the teams in dockerizing their approaches (frequent issues were files that were not executable in an image, or resources like libraries or models that were not included in the image). eventually, all teams who wanted to make a software submission actually submitted software that produced valid outputs. for the execution, participants could select resources out of four options: (1) 1 cpu core with 10 gb ram, (2) 2 cores with 20 gb ram, (3) 4 cores with 40 gb ram, or (4) 1 cpu core with 10 gb ram and 1 nvidia geforce gtx 1080 gpu with 8 gb ram. upon request, tira could even execute a software on a kubernetes cluster with 1,620 cpu cores, 25.4 tb ram, and 24 geforce gtx 1080 gpus. a team could run their software as often as they wanted using different resources to study the scalability and reproducibility (e.g., does a software run on a gpu yield the same results as on a cpu?). ## task description and datasets in the pan clickbait challenge at semeval 2023, we follow our previous observation @xcite and distinguish three different types of spoilers: (1) short phrase spoilers, (2) longer passage spoilers, and (3) multi-part spoilers. examples are given in figure 1 . based on the hypothesis that tailored spoiler generators for the different spoiler types will be the most effective (i.e., identifying the required spoiler type before actually generating a spoiler might be beneficial), we organized two subtasks: (1) spoiler type classification, and (2) spoiler generation with given spoiler type information. the input for spoiler type classification is the clickbait post and the linked document. the task is to identify whether the clickbait post requires either a phrase (the upper post in figure 1 ), a passage (the middle post in figure 1 ), or a multi-part spoiler (the bottom post in figure 1 ). for an input post (with a uuid), an output in the form of {"uuid": "<@xmath0, "spoilertype": "<@xmath1, @xmath2, passage, or multi. the input for spoiler generation is the clickbait post, the linked document, and the required spoiler type (phrase, passage, or multi). for an input post (with a uuid), an output in the form of {"uuid": "<@xmath3, "spoiler": "<@xmath4, @xmath5. ## task 1: spoiler type classification the goal of the first task of the pan clickbait challenge at semeval 2023 was to classify which spoiler type a given clickbait post requires. we explored a wide range of approaches during a pilot study to provide a competitive baseline as the basis for the shared task. we evaluated the effectiveness of spoiler type classification using the balanced accuracy over all three classes as the main effectiveness measure that we complement by the precision, recall, and f1 score in detecting all three classes. overall, 25 teams submitted results to task 1. ## task 2: spoiler generation the goal of the second task of the pan clickbait challenge at semeval 2023 was to generate the actual spoiler that could be shown to users to satisfy the curiosity introduced by a clickbait post. we explored a wide range of approaches during a pilot study to provide a competitive baseline as the basis for the shared task. we evaluated the effectiveness of spoiler generation by contrasting bleu-4, meteor, and bertscore scores (generated vs. ground-truth spoiler). overall, 17 teams submitted results to task 2. ## post-hoc reproducibility experiments we used tira @xcite as submission system so that we can publish all artifacts resulting from the shared task to improve reproducibility and re-usability. during the shared task, tira maintained all data in a git repository, where each software execution and evaluation were triggered by commits in continuous integration and continuous deployment pipelines. we publish this repository, including all datasets, results, evaluations, logs, metadata, and software snapshots. 9 listing 1 exemplifies how software submissions can be re-executed on the same or new data (the api allows the re-use of existing datasets but also new data of the same structure, requiring only docker and python 3 as dependencies). consequently, the shared task repository that we published after the completion of the shared task now serves as a entry point for follow-up studies. all researchers can fork this repository and contribute. ## conclusion the second pan clickbait challenge hosted as task 5 at semeval 2023 received submission from 30 active teams on two subtasks: (1) spoiler type our results show some substantial advancements, but there is still much room for further improvements. first, as no spoiler generator achieves the highest effectiveness across all three spoiler types, the best generators still complement each other and combining the underlying ideas can probably further increase the effectiveness. second, as the most effective spoiler generator is ensemble-based and comes as a 45 gb docker image, distilling a model with a better efficiency-effectiveness tradeoff also is an interesting direction for future work.