S am Miller at S em E val-2023 Task 5: Classification and Type-specific Spoiler Extraction Using XLNET and Other Transformer Models (2023)

Abstract: This paper proposes an approach to classify andan approach to generate spoilers for clickbaitarticles and posts. For the spoiler classification,XLNET was trained to fine-tune a model. Withan accuracy of 0.66, 2 out of 3 spoilers arepredicted accurately. The spoiler generationapproach involves preprocessing the clickbaittext and post-processing the output to fit thespoiler type. The approach is evaluated on atest dataset of 1000 posts, with the best resultfor spoiler generation achieved by fine-tuninga RoBERTa Large model with a small learningrate and sample size, reaching a BLEU scoreof 0.311. The paper provides an overview ofthe models and techniques used and discussesthe experimental setup.

## introduction in advertising and social media, there are common patterns and tactics that are employed to capture and capitalize the attention of consumers. one of those is the use of so-called "clickbait", which create curiosity in the users that motivate them to click on the articles @xcite , @xcite . the proposed solution, therefore, is to generate an approach that automates the generation of a spoiler, that spoils the information needed to close the users curiosity gap. the user can then make a more informed decision whether they want to proceed. the clickbait posts and articles are mostly in english language. the task is split into two parts, in the first task, the type of the required spoiler is determined, whether it's a shorter phrase, a longer passage or a multipart spoiler. in the second part, the actual spoilers will be generated or extracted. given task 2, our approach was to fine-tune a question answering model and use post-processing the respective spoiler type, so that the format and length of the answer fits the requested spoiler type. for multipart spoilers, recursive spoiler generation was used to generate multiple answers. furthermore, the possibility of doing preprocessing in the form of query rewriting was explored. we tried different pretrained models, and larger models with more pretraining (roberta large squad-2) outperformed smaller models significantly. our result for task 2 was comparable to the other groups, but worse than the baseline (0.311 bleu on our approach compared to 0.38 on the transformer baseline @xcite ). models were added to the docker images @xcite to ensure replicability. ## background the tasks specified input and output formats of the software. as input, a json file with several features for the respective tasks was given. all paragraphs were separated in the input files. the output file must also be a json file, while the format depends on the task. for task 1, it must contain the uuid and the predicted spoiler type. for task 2, it must contain the uuid and the spoiler as a string. the train dataset was used to refine our approaches, and the validation dataset was used to test our approaches. the evaluation was generally performed via docker images and tira 1 . the bleu score @xcite was calculated on the tira service @xcite . for task 1 and 2, the best performing images can be accessed via the link given in the appendix. ## experimental setup for task 1 and 2, the given train and validation datasets were used to train and test the models. we fine-tuned models for both tasks on the training dataset provided by the sem-eval organizers and then tested it locally on subsets of the data, before submitting it with the use of a docker @xcite image to be run against the validation dataset on the tira system @xcite . task 1 since task 1 is a simple text classification problem, the model is fine-tuned by preprocessing the input data in a way xlnet understands it, which includes special tokenizing, adding separators. xlnet does not allow for non-numerical labels, so the labels got encoded into either 0, 1 or 2. the total calculation time per training iteration comes down to a little over 14 hours. although additional information could have been used, the only text to predict the label was the "posttext". task 2 for task 2, roberta @xcite and distilbert @xcite were used with a question answering head. these models were fine-tuned using different hyperparameters. to prepare the clickbaits for the questionanswering model and to optimize the output of the spoilers, a number of pre-and post-processing techniques were applied. for the preprocessing, the clickbait was first converted to lowercase and tokenized using spacy (honnibal and montani, 2017 ) foot_0 . if the clickbait contained an interrogative word (such as "how", "what", "when", etc.), it was moved to the beginning of the clickbait. additionally, to handle interrogative words like "how much" or "how many", a special condition was added. a transformer question-answer pipeline @xcite foot_1 from was used to obtain spoilers for all three spoiler types. for phrase spoilers, a post-processing function was implemented to limit the returned answer to a maximum of 5 words/tokens. two post-processing approaches were implemented for sentence spoilers. the first procedure removed the entire sentence if the predicted response contained less than 5 words/tokens, and repeated the process until a suitable response is obtained. the second method was to return the whole sentence that contained the predicted answer if the predicted answer contained at least 5 words/tokens. in the case of multi-part spoilers, a spoiler was predicted, then the sentence that contained the spoiler was removed, and the pipeline was run again. this process was repeated until a total of 5 spoilers had been obtained. ## conclusion task 1 shows how strong large language models have become. even by fine-tuning them by a small degree, they prove to be very strong in predicting and classifying text spoilers. the results from the second task also show that larger models such as roberta outperform smaller models like distilbert. additionally, using preprocessing techniques like query rewriting and post-processing based on spoiler type improved the quality of the generated spoilers. for future work, the approach could be extended to handle additional spoiler types or incorporate other preprocessing techniques by using more advanced text normalization techniques. additionally, exploring the use of multi-language models could improve the performance on non-english posts.