NL 4 IA at S em E val-2023 Task 3: A Comparison of Sequence Classification and Token Classification to Detect Persuasive Techniques (2023)

Abstract: The following system description presents our approach to the detection of persuasion techniques in online news. The given task has been framed as a multi-label classification problem. In a multi-label classification problem, each input chunkin this case paragraphis assigned one of several class labels. Span level annotations were also provided. In order to assign class labels to the given documents, we opted for RoBERTa (A Robustly Optimized BERT Pretraining Approach) for both approachessequence and token classification. Starting off with a pre-trained model for language representation, we fine-tuned this model on the given classification task with the provided annotated data in supervised training steps.

## introduction political rhetoric, propaganda, and advertising are all examples of persuasive discourse. as defined by @xcite , persuasive discourse is the nonreciprocal "attempt or intention of one party to change the behavior, feelings, intentions, or viewpoint of another by communicative means". thus, in addition to the purely content-related features of communication, the discursive context of utterances plays a central role. semeval-2023 task 3 @xcite considers persuasion as a communication phenomenon. with this approach, it is assumed that communication depends not only on the meaning of words in an utterance, but also on what speakers intend to communicate with a particular utterance. this concept is from the linguistic subfield of pragmatics. it is not always possible to derive the function of an utterance from its form and additional contextual information is often needed. recent research like @xcite ) @xcite indicates the possibility that transformer-based networks capture structural information about language ranging from syntactic up to semantic features. beyond these features, these architectures remain almost entirely unexplored. this task poses an attempt to explore the limits of the prevailing approach, in particular, investigating transformers ability to capture pragmatic features. ## background the central focus of this assignment is manipulative persuasion. the related task is to identify and evaluate propaganda and persuasion as found in social media. to identify and characterize manipulative persuasion, the context can be stretched arbitrarily far across aspects of epistemology, logic, intent estimates, psychological biases, knowledge of pre-existing narratives, and even physical context. however, to potentially solve this problem in an automated fashion, the prevailing method is to frame the given task as a classification problem. the different propaganda methods are understood as distinguishing criteria. documents or even subsections of documents are annotated based on these features and thus form the input for training machine learning models, which should then be able to automatically recognize and classify corresponding sections. the undeniable success of this approach for many applications (for example, in ner) is due to the fact that the required features arise directly from the data or are already captured in the data representation used (word embeddings). in fact, current word embeddings already contain representations of a wide range of syntactic and morphological features that can be used to solve many problems. in the following pages, we discuss whether and to what extent the required characteristics are reflected in the training data. in particular, we consider whether and to what extent linguistic structures can be used as a decision criterion. in explaining our findings, a pragmatic perspective is adopted. in general, descriptive, analytical, and linguistic approaches such as speech act theory and rhetoric (or the use of specific rhetorical devices) are used to characterize public (political) discourse. referring to the speech act theory @xcite , linguistic features, also in their form as rhetorical features, are assumed to be identified in the locutionary act. the illocutionary and perlocutionary acts, meanwhile, involve more complex information that might be used in feature engineering, thus incorporating the dimension of discourse. ## system overview in this study, we evaluate and compare two different approaches: sequence classification and token classification. the comparison is performed at the level of trained models on the same set of data. the different scoring paradigms arise from applying token classifier and sequence classier heads, respectively, on a pre-trained model as the base model. we suggest that contextual information is captured and processed differently in both approaches, leading to a qualitative difference in the scores. our results show that sequence classification is superior. ## experimental setup in both cases, fine-tuning was done with an nvidia tesla v100 gpu using the pytorch @xcite framework with a vocabulary size of 50265 and an input size of 512. the model was trained to optimize the objective for 50 epochs. ## results we participated in the persuasion techniques detection task and focused on the english dataset. official evaluation results on the test set are presented in our submission is based on the trained model after 100 training epochs in the case of the multi-label sequence classification. we were able to improve the official f1-micro score of 0.30761 to 0.34280 by evaluating the intermediate checkpoints. based on this difference, we can assume a significant overfitting in our resulting model. with a maximum f1-micro score of 0.31325 in the case of the token classification approach, results remained in all cases below those of the sequence classification approach. compared to the sequence-level assignment, it seems that token-level assignment requires an increased discriminatory power between the individual categories, which is clearly not sufficiently satisfied in this case. when improving on the pretrained baseline models, class imbalance appears to be a primary challenge. with a highest ranked f1-micro score of 0.37562, it is necessary to discuss other causes for the low discriminatory power. possible challenges related to neural architectures arise either from under-specification of the objective function or from general difficulties of feature engineering. difficulties with the objective function arise when the target variables, in our case the individual persuasion techniques, conceptually cannot be well separated. issues with feature engineering are to be expected when required features cannot be captured from the training data. @xcite suggest that transformer-based networks are able to glean structural information-both syntactic and semantic-from language. if this is so, we expect that further important features may be hidden in the broader context, especially when it comes to manipulative communication. since these features do not emerge from the training data, they must be made available to the training process in some other way. features of interest may be derived from research in pragmatics.