Team JUSTR 00 at S em E val-2023 Task 3: Transformers for News Articles Classification (2023)

Abstract: The SemEval-2023 Task 3 competition offers participants a multi-lingual dataset with three schemes one for each subtask. The competition challenges participants to construct machine learning systems that can categorize news articles based on their nature and style of writing. We esperiment with many state-of-the-art transformer-based language models proposed in the natural language processing literature and report the results of the best ones. Our top performing model is based on a transformer called “Longformer” and has achieved an F1-Micro score of 0.256 on the English version of subtask-1 and F1-Macro of 0.442 on subtask-2 on the test data. We also experiment with a number of state-of-the-art multi-lingual transformer-based models and report the results of the best performing ones.

## introduction transformer-based models have seen immense success in natural language processing (nlp) tasks and the entire machine learning field in general. the self-attention mechanism combined with a deep number of layers has proved to outperform traditional deep learning architectures like the recurrent neural network in the nlp field. transformer models have also allowed for the process of transfer learning of large models that were pretrained on large corpora. this in turn gave nlp researchers the ability to use the state-of-the-art pretrained models on downstream nlp tasks which has been proven to be a much better method than only training on the downstream task data. in addition to their success in nlp tasks, transformerbased models have also been widely used in computer vision and speech recognition applications. one key advantage of transformer models is their ability to capture long-range dependencies in the input data through self-attention, allowing them to make more informed predictions about the relationship between different elements in a sequence. this has led to breakthroughs in tasks like language translation, sentiment analysis, and questionanswering. the pre-trained transformer models like @xcite and bert @xcite have achieved impressive performance in many nlp benchmarks @xcite , often outperforming human-level accuracy @xcite . another advantage of pretrained transformer models is their ability to adapt to new domains with minimal training data, which greatly reduces the cost and time required for developing new nlp applications. the semeval-2023 task 3 competition @xcite is an nlp task in that focuses on the detection of various categories in news articles. our research aims to explore the performance of different transformerbased models on the task at hand and provide insights into the strengths and weaknesses of these models for detecting various types of categories in news articles. ## dataset the semeval-2023 task 3 competition offers datasets of labeled articles from 6 languages. the languages are english, german, italian, french, polish, and russian. each language has a dataset for training, validation and testing. the competition also offered testing datasets for 3 "surprise" languages (spanish, greek and georgian). in our work, we focus only on the english, german, italian french and polish languages. each dataset was annotated in three schemes one for each sub task. subtask-1 titled "news genre categorization" focuses on categorizing the article based on whether an article is an "opinion", a "report" or a "satire" piece (multi-class classification). subtask-2 is titled "news framing" and aims to detect the presence of a set of 15 frames that would potentially be used in an article (multi-label classification). finally subtask-3 is titled "persuasion techniques detection" and aims to detect the presence of a set of 23 pre-defined persuasion techniques in a single paragraph (multi label classification). subtasks-1 & 2 work on an article level while subtask-3 works on the paragraph level. the datasets are highly imbalanced where certain labels are more frequent than others. for example, the training dataset of subtask-1 for the english language has 382 articles labeled as "opinion", 41 articles as "reporting" and only 10 as "satire". similarly the labels of subtask-2 for the english dataset are imbalanced with classes such "political" and "morality" having a high number of occurrences @xcite while other classes such as "public opinion" and "economic" have low occurrences @xcite . this class imbalance represents one of the major challenges these datasets pose since it would be difficult for the models to learn how to detect the classes with low frequencies in the training data. figure 1 shows a bar plot of some of the training datasets for subtask-1 showcasing the class-imbalance across all languages. another major challenge is the inability of the models to take an entire article as input whether in training or inference. the transformer based models use a fixed maximum number of input tokens configuration with 512 tokens being a common one. the articles in the training data typically have a number that exceeds 512 tokens with 75% of the articles having more than 1300 tokens. sequences that exceed the maximum number of tokens are truncated to only include the first n amount of tokens in an example where n is the maximum input length of a transformer model (e.g. 512). ## methodology in this section we describe our methodology where we experiment with a number of different models and report the results. we only focus on the first two subtasks "news genre categorization" and "framing detection". for both subtasks, we stack a fully connected linear layer that takes as input, the pooled output of the transformer model. the pooled output is a vector that represents the input sequence encoded by the transformer. the activation function of the linear layer depends on the subtask. for subtask-1 which is a multi-class classification problem we use a softmax activation. while for subtask-2 which is a multi-label classification problem we use a sigmoid activation function. for all our networks we use the adam optimizer with weight decay and a dropout of 0.1 on the last pooling layer of the transformers. ## conclusion we have discussed the transformer-based models that we experimented with. our best model that we fine-tuned was longformer and had surpassed the other models due to its ability to read higher sequence lengths. while it is intuitive that a larger max input length leads to better performance, it is possible that adding a preprocessing pipeline instead of naively training on the news articles with no pre-processing would yield better results. the preprocessing pipeline could consist of either handling the class imbalance or focusing more on finegrained analysis rather than processing the entire all at once articles.