Stephen Colbert at S em E val-2023 Task 5: Using Markup for Classifying Clickbait (2023)

Abstract: For SemEval-2023 Task 5, we have submitted three DeBERTaV3[LARGE] models to tackle the first subtask, classifying spoiler types (passage, phrase, multi) of clickbait web articles. The choice of basic parameters like sequence length with BERT[BASE] uncased and further approaches were then tested with DeBERTaV3[BASE] only moving the most promising ones to DeBERTaV3[LARGE]. Our research showed that information-placement on webpages is often optimized regarding e.g. ad-placement Those informations are usually described within the webpages markup which is why we conducted an approach that takes this into account. Overall we could not manage to beat the baseline, which we lead down to three reasons: First we only crawled markup for Huffington Post articles, extracting only p- and a-tags which will not cover enough aspects of a webpages design. Second Huffington Post articles are overrepresented in the given dataset, which, third, shows an imbalance towards the spoiler tags. We highly suggest re-annotating the given dataset to use markup-optimized models like MarkupLM or TIE and to clear it from embedded articles like “Yahoo” or archives like “archive.is” or “web.archive” to avoid noise. Also, the imbalance should be tackled by adding articles from sources other than Huffington Post, considering that also multi-tagged entries should be balanced towards passage- and phrase-tagged ones.

## introduction the shared task "clickbait challenge at semeval 2023 -clickbait spoiling" on pan was about classifying and predicting spoilers from english (gossip) articles @xcite provided via tira @xcite . it poses several technical challenges, especially handling text data with a high degree of diversity in genre, writing style, and structure. the task aims to close the curiosity gap clickbait posts cause within their readers by providing an informative summary. solving this problem would benefit the use of social media since click baits are an annoying phenomenon aiming to manipulate one into visiting a page which may affect the user's credibility and quality perception in a negative way @xcite . the task consists of two subtasks: our system focuses on solving the classification task with a fine-tuned debertav3 large model trained on the given webis clickbait 2022 corpus, which we enriched with information from markup crawled from the original websites. we have chosen this approach since we discovered that (gossip) articles, in most cases, as webpages, are subject to some professional style guide aiming to make visitors stay on the page, which is essential, especially when it comes to advertising revenues @xcite . the results fell short of expectations since we only tested with markup from huffington post articles and only from the training dataset. nevertheless, out of 31 submitting teams for task 1, our team, for most measures, got among the top 10 teams and the top 3 with the highest f1 score in classifying passage spoilers. ## background with the webis clickbait 2022 corpus, the dataset contains 4000 clickbait articles, split into training and validation sets. for solving the classification task, we focused on the following: we utilized the state-of-the-art transformerbased architectures bert @xcite and debertav3 @xcite , which, by adding a disentangled attention mechanism and an enhanced mask decoder, significantly improves performance on various tasks compared to bert ( he et al., 2021) our experiments show that taking markup into account regarding webpage analysis is quite common in visually rich document understanding. however, since the given dataset does not meet the requirements to utilize models like markuplm @xcite or tie @xcite , we had to come up with our approach. due to the small size of the training dataset, containing only 3200 rows, we also used the validation dataset for finetuning, which consists of 800 rows, to improve accuracy on every run. ## system overview we initially started with fine-tuning bert base uncased to gain fast insights into how parameters influence its accuracy. because of the different hardware setups available to us (see ## experimental setup we evaluated three transformer-based models (for specifications, see for our markup-based approach, we crawled the original articles using beautifulsoup and cleared the results with a simple regex pattern (see section dataset enrichment with markup for a more detailed description) we submitted three debertav3 large models to tira, fine-tuned on the validation set, the validation set containing only 1 16 of all huffington post articles and the validation set enriched with markup for huffington post articles. ## results as shown in since we had many mixed results, we recommend further balancing the dataset, especially to compensate for the overrepresentation of huffington post articles and the underrepresentation of multitagged articles. since our markup-based approach resulted in the lowest accuracy, we assume this is because we only considered articles from huffington post. also we only managed to focus @xmath0. since with markuplm and tie, there are high-performing models to use for domains like webpages to solve prediction and classification tasks, we strongly recommend taking this into account for annotating clickbait articles in the future, always saving the pages' entire markup within the dataset and avoiding embedded sources like "yahoo" or archives like "archive.is" since they build markup around the embedded article and might leave out content like an advertisement which could hint to the position of the spoiler in the text. ## conclusion we submitted several approaches, utilizing debertav3 large to solve the classification task of semeval-2023's task 5. although we could not quite reach the baseline's accuracy we could point out weaknesses within the given dataset caused by the overrepresentation of huffington post articles and the underrepresentation of multi-tagged entries. also, it misses providing the original markup, which we tried to crawl retroactively after we figured out that design is one of the critical parts of information placement on the website. this approach was not very successful, which might be because