Brooke- E nglish at S em E val-2023 Task 5: Clickbait Spoiling (2023)

Abstract: The task of clickbait spoiling is: generating a short text that satisfies the curiosity induced by a clickbait post. Clickbait links to a web page and advertises its contents by arousing curiosity instead of providing an informative summary. Previous studies on clickbait spoiling has shown the approach that classifing the type of spoilers is needed, then generating the appropriate spoilers is more effective on the Webis Clickbait Spoiling Corpus 2022 dataset. Our contribution focused on study of the three classes (phrase, passage and multi) and finding appropriate models to generate spoilers foreach class. Results were analysed in each type of spoilers, revealed some reasons of having diversed results in different spoiler types. “passage” type spoiler was identified as the most difficult and the most valuable type of spoiler.

## introduction according to the clickbait spoiling generate research paper @xcite @xcite , clickbait spoiling generation is similar to the task of machine reading comprehension (mrc), where context is the full article/advertises with its title waiting for a reader. question of mrc corresponds to the clickbait, which is a short text that arouses curiosity without providing much informative summary, aiming at attracting people to click into the link towards the context article. and the answer is a spoiling text that could satisfy people's curiosity raised by the clickbait, therefore people could save the time of reading. detailed research about the task and its dataset has been conducted in the referring paper. in order to spoil its content, we are listing a couple of key points already verified by the paper and its dataset, and they would be considered as important priors in our experiments. during data processing some flaw on target span positions were found, target fixing procedure was applied, details would be described in section3.1. due to this target fix procedure, direct result comparison between this paper and the previous research paper may not be appropriate. on the other hand, "multi" type spoilers are included in our experiment. within a multi example, multiple text spans that are often locate in different paragraph lines of context. considering the differences in dataset, training tasks and the starting pretrained model, we designed different features, module layers, losses and postprocesses for each type of spoiler. experiment results verified that three types of spoilers shall be trained separately with it's own parameters. using a classifier with accuracy 0.76, the overall benefit of separation training overcomes the drawback of misclassifing the spoiler and send it to a wrong spoiler extractor. there existed a distribution shift between validation and test dataset, due to a small dataset size and the random dataset splitting. consistant gap of the measures between validation and test dataset were observed. the reason of observed test performance being much better than validation would be discussed. evidences and error analysis suggested "passage" type spoiler and bleu-n may be the causes of troubles. experiment results showed mixing all types of spoiler in a training dataset would give lower measures on both bleu-n and meteor than firstly predicting the spoiler type, then use separate models to extrace spoilers. replacing the classifier with a hypothetical perfect classifier, would produce the best measures. separating the spoilers would improve on the signal-noise ratio, which would be shown as a very important aspect of explaining performance differences between models. the deberta qa model had been pretrained for squad, and was used as initial parameters of my models. relative paragraph id was used as an additional feature for classifier and "multi" model. contrastive learning losses were used for "passage", "multi" and "mix" models, because their span token lengths were larger in both mean and variance. learning the start and end boundary of a longer spoiler, is less similar to a typical squad answer, whereas boundary of a short spoiler (i.e. phrase) typically encloses an entity, which is more similar to a squad answer. more modifications were applied to "passage" and "multi" models due to their smaller similarity answers towards squad. ## background table1 shows a clickbait-spoiler data example, context text is a concatenated string of title and paragraphs, clickbait is a question-like sentence, and its answer is the spoiler. the length of each paragraph is at regular sentence length, but the joined context could contain over 2k tokens, which is a length exceeding the typical max length of transformer encoders. spoilers could appear anywhere in the context, the position of one spoiler span can be expressed in two ways: dataset paragraph spans has to be converted into context spans, then the context would be splited into shorter texts, called feature texts. each feature text has a fixed number of tokens overlapping with its neighbour feature texts, in order to reduce the likelihood of having a ground truth span that starts and ends across two consecutive feature texts. the feature texts belonging to an example, are then appended after the example clickbait text, forming the complete input texts, like shown in fig1 the pretrained deberta qa model predicts the start and end position logits separately and by adding the logit scores s @xmath0, the span score which starts at token position i and ends at token position j can be calculated. selecting the span with the maximum score gives the answer span. if the feature does not contain any answers, the [cls] token position shall have the maximum answer, i.e. s 00 > s ij ∀@xmath1, @xmath2. squad dataset was used to pretrain the baseline model, and my models are all fine tuned starting with this baseline version. 2 in order to merge the results of all features belonging to a single example, a beam search mechanism was used in the baseline postprocess. pseudocode can be used to describe the postprocess alg1: this postprocess works fine for single span results, but it need to be modified for "multi" spoilers. table2 is the results of the mix model on validation dataset after trained, with detailed bleu-n and meteor on each spoiler type. the performances indicates phrase spoilers are the most similar texts towards a typical qa answer, whereas the multi spoilers have the least similarity. multi spoilers cannot be produced in the mixed model, it only predicts one answer span. a classifier was trained, then grid search of confidence threshold of classifier was performed. by comparing the classifier's predicted spoiler type confidence with the thresholds, we could decide whether a type-specified model should be used, and those examples with low type confidences goes into the mixed model. 5 models were trained, three for each individual type, one for mixed, and one classification model. the three type-specific models all output a score matrix s ij , storing the reranking scores of spans start at token position i and ends at token position j. their input features, prediction heads, and postprocess are different, depending on their ground truth similarity with the original qa task, more modifications were applied if similarity is low. multitask training setup was considered but not experimented due to the gpu memory was limited within 24g, so encoder layers do not share parameters. ## experimental setup differences of setups for training all the models are listed in table3. rows are the model options, including: 1) absolute token positions in each paragraph; 2) paragraph ids used to calculate relative paragraph embedding and the antisymmetric modifier; 3) multi-softmax in cross entropy; 4&5) weights of cross entropy loss and contrastive loss; 6) width of margin in terms of number of tokens when defining the whitelist; 7) max number of tokens of a single answer span allowed; 8) top n number during postprocess; 9) number of answer sapans after grouping similar answers. ## results type classification result is shown in table5. reason of such a big difference is due to the imbalanced "passage" examples in the two dataset, and we would discuss it later with more details. we would show test and validation results together as the evidences. classifier val acc 0.7625 test acc 0.7925 using a classifier already trained, confidences of each spoiler type can be obtained. a grid search on validation dataset of thresholds for "phrase", "passage" and "multi" was performed. if an example has its top1 type probability above the confidence threshold of the corresponding type, then the typespecified model would extract the answer, otherwise the "mix" model would extract the answer instead. the large difference of performance between validation and test datasets was mainly due to the distribution shift. validation dataset has a larger proportion of "passage" spoilers, therefore increasing the difficulty of making accurate type predictions. if we remove the thresholds and analyse the confusion matrix without the "mix" model, we could discover that both "phrase" and "multi" are most likely to be misclassfied to "passage". both "phrase" and "multi" examples had a proportion of 23% which were predicted as false "passage", whereas only 12% passages were predicted as false "phrase" or "multi". this implied "passage" data is naturally noisy in the sence that some of its examples showed moderate similarities with the other two types. unlike the "phrase" and "multi" examles which showed good orthogonality in terms of type similarity, "passage" is more like a "mixture" of all types of spoilers. some "passage" had a short spoiler text which is shorter than the mean length of "phrase" spoilers, therefore it is reasonable for the model to misclassify "phrase" and "multi" to a "passage" if confidence level is low. that explains why we need high confidence thresholds for "phrase"(0.75) and "multi"(0.7), these two models show much better meteor performance compare with "passage" model when they are extracting their own type of spoilers. in table6 we could conclude the "passage" had a similar noise level as the "mix", because "passage" had performances which were closest to "mix" model. low confidence threshold of "passage"(0.45) reflected the fact that "passage" model was accepting the "noisy" (or hard) examples, and it was still a slightly better model compre to the "mix" in most cases. comparing the number of all types of spoilers in validation and test dataset, we could exaplain why test results are better than validation results in nearly all the tables, because validation has 170 passage spoilers out of 400, whereas test has a fewer number of 152. even though best snapshots of models on the validation dataset was chosen, the deterioration of performances due to the distribution shift of having more "passages", is still prominent. spoiler type prediction accuracy confirms the fact that validation dataset was harder to be correctly classified due to these extra "passage" examples. we have explained the causes of bad performances of "passage" model on validation dataset, so bad case analysis was done on the errors. fig3 showed the detailed distribution of measure metrics of examples predicted as "passage" type. the type labels in the figure are the ground truth types, the x-axis are measure metrics, y-axis are the example counts. hence the primary reason of bad performance of "passage" predictions in table7 can be identified as wrongly classifying "phrase" and "multi" examples to a false "passage". table10 showed the bleu-n and meteor of predicted spoilers with the "passage" model for each ground truth spoiler type on validation dataset. when ground truth type is "passage", the results were close to the corresponding results in table6, which are the correctly classified results. however if the ground truth type is "phrase" or "multi", their results would be significantly lower than the results in table6. summing the true and false "passages", the predicted bleu-n and meteor in table7 would be produced, giving the worst performance model which was the passage type model. fig3 also revealed the fact that bleu-n is not as smooth as meteor. the term "smooth" means the histogram of the metric is a more even distribution, rather than a polarized distribution, which has concentrated measure values at the low end [0,0.1] or the high end (0.9, 1.0]. this means bleu-n is encoding less information of similarity between ground truth and predicted spoiler, because it has a lower entropy. human analysis of differences between bleun and meteor was done by sorting ∆ = abs (meteor -bleun ) in descending order for the "passage" predictions of validation dataset. ta-ble4 showed top5 examples that the two metrics showed a largest divergence. the predicted passages tend to include the ground truth as a substring, and it is reasonable to penalize the tedious unwanted texts, but there are cases like "talk about their money" and "handling their money and talking about it with their partners", which convey essentially the same information only with slightly different way of saying. silly mistakes for both metrics happens in short phrase spoilers. meteor failed with single-token predicted and target spoiler, for an example text @xmath3, has bleu-n 1.0 and meteor 0.5. this directly caused blue-n larger than meteor value in phrase ex-amples. bleu-n failed with substrings such as text @xmath4, california" and text @xmath5, which has bleu-n 0.0 and meteor 0.4. non-english names also failed for both measures, such as text @xmath6, producing bleu-n 0.0 and meteor 0.25. these are common mistakes made by metrics, the predictions are not absolutely wrong. extracting a short phrase or passage from a long article would always encounter such a problem that a spoiler could have "isotops" which bring the same information but expressed differently. largest common string was used during dataset repair, maybe it is also a good metric for short texts. bert score may also be able to reduce these pseudo mistakes. it would be interesting to make a further study on designing a suitable metric. because of the deadline time and the gpu memory limitation, multi-task setup was considered but not experimented, it is reasonable to hypothesize that sharing the first few layers of encoder could be benifitial, further study would also be focused on this approach. ## conclusion clickbait spoiling generation can be regarded as an mrc problem, its model accepts similar input features as squad. the spoiler answer types and the article paragraph layout information were important because these are newly added components towards a typical qa problem. designs considering multi-paragraph answers produced the largest improvement in the "multi" model results using the mix as a baseline, this can be concluded by comparing table2 and table6, revealing improvements on bleu-n for phrase, passage and multi are 0.022, 0.115 and 0.285 respectively, and the improvements on meteor are 0.052, 0.093 and 0.159. separating different types of models brings improvements in all models. despite having the least similarity with the starting qa task and the least number of training examples, multi type spoiler extractor achieved the largest improvement. phrase type spoiler extraction is the easiest task, it has little improvement after the type splitting, and in most cases the absolute performance is the best among the all types. it is the most similar task as the starting pretrained task, experiments of adding extra input features, loss design or postprocess showed little influence on its performance. the influences are too small to be distinguished within a 400 example 1-fold validation, k-fold experiments shall be done to make a convincing weak improvement, making phrase type spoiler task not as attracted as the other types. passage type spoiler is the most difficult in the sence that its data is the noisiest. there is not a clear feature indicating the spoiler should be a "passage" type instead of the other two types, and there is not a good general description of the passage ground truth texts. it is not a named entity, not a list of enumeration items, not a complete sentence, but it is any possible substring of the context. therefore we encounter the most unknown inputs and outputs, results of passage model is the closest towards mixtype results, confirming the randomness of passage spoilers. since passage type has a relative large proportion in the dataset and a relative poor performance, more study effort was put on passage model. fig3 revealed the most important drawback for the passage model is misclassifing spoiler type and use it to predict a "phrase" or "multi" example. tuning the thresholds showed a trade-off between task generalization and specialization. if a model is good at a specific type of task, then as shown in table10, its performance in a general task (where passage is the closest to the general mix) would be bad, hence we need a high threshold. designing the training for a general task like passage spoiler extraction, is challenging and valuable, further study would be focus on this. bad case study also revealed some fails for both bleu-n and meteor for short spoilers with only a few tokens. for longer spoiler texts, changing the way of expressing and difference in degree of penalty of substring, are the two major causes of diversified results between the two measures. a measuring metric to bring the gaps closer shall be studied in future.