John-Arthur at S em E val-2023 Task 4: Fine-Tuning Large Language Models for Arguments Classification (2023)

Abstract: This paper presents the system submissions of the John-Arthur team to the SemEval Task 4 “ValueEval: Identification of Human Values behind Arguments”. The best system of the team was ranked 3rd and the overall rank of the team was 2nd (the first team had the two best systems). John-Arthur team models the ValueEval problem as a multi-class, multi-label text classification problem. The solutions leverage recently proposed large language models that are fine-tuned on the provided datasets. To boost the achieved performance we employ different best practises whose impact on the model performance we evaluate here. The code ispublicly available at github and the model onHuggingface hub.

## introduction semeval task 4 "valueeval: identification of human values behind arguments" has as a goal to classify a textual argument across one or more human value categories @xcite . understanding and identifying human values in arguments is a difficult task as they are implicit and their definitions are often vague. at the same time they are studied in various domains like social studies and formal argumentation. as a result, having a system aiding to the task could have a measurable impact on such studies. to build an nlp system that classifies input text data among human values we build on recent work on large language models (llms). these models are capable of "understanding" english text and our hope is that they can be used to achieve satisfactory performance on the task. the valueeval input data consist of three pieces of information: on the other hand, the prediction step is to recognize the human value for each input among 20 values. this is a multi-class, multi-label classification problem as most inputs belong to several human values. in our experiments we discovered that llms are a suitable solution to the task. properly encoding the input information for the llms and fine-tuning the models with the data provided by the organizers quickly improves the performance. also, we found that bigger models achieve better results if tuned properly. adding data to the training set also has a big impact on the obtained performance. we tried various ideas in the prototyping phase: from different ways to encode the model input to various architecture choices on how to use the model output. compared to more traditional machine modeling lifecycle where a lot of effort is put on feature engineering, we often realized that while some feature engineering benefits the performance using smaller models, its impact on larger model is diminishing. the best system of the john-arthur team is currently ranked at position 3 of the private leaderboard. in terms of teams, john-arthur is ranked 2nd, behind the adam smith team, who submitted the 2 best performing systems. we make the code that trains a model publicly available foot_0 and the model of the best submission also available. foot_1 2 background task 4 of semeval 2023 consists of a single track. task submissions were handled using the tira platform @xcite . the dataset of the challenge is described in @xcite . an example input to be classified is as follows: this is framed as a multi-label, multi-class text classification problem. the evaluation measure the organizers chose is the macro-averaged f 1 score. ## system overview we model the problem as a text classification problem and we intend to use llms on the premise that they can uncover and model the implicit semantics of the input data to be able to successfully predict the human values. ## experimental setup the organizers released several data splits. the main dataset comprised 3 splits (train, test, validation) that consist of 5,393, 1,897 and 1,576 data points respectively. also, 100 more labeled data points were released as "arguments-validationzhihu.tsv" from related work. in early iterations we found that the model greatly improved with more training data. we discovered this by observing a lift across all model architectures when adding as few as 100 extra data points in the training data ("arguments-validation-zhihu.tsv") which were few compared to the 5,393 data points of the training dataset. motivated by this observation that our systems benefits a lot from extra training data, we decided to submit two runs with the same modeling and training configurations. the first run uses as training data the training data and the "argumentsvalidation-zhihu.tsv" dataset while the second run uses as training data the original training and validation datasets (5,393 + 1,897 data points) and as validation the 100 points of "arguments-validationzhihu.tsv". we opted for having a validation set because we wanted to adhere to best neural network modeling practises and monitor the evaluation loss and the evaluation metrics to ensure that the model trains properly. we leave using the concatenation of all available data for training as well as performing some form of data augmentation or distant supervision using unlabeled data as promising directions of future work. ## results ## conclusion in this paper we presented the submission of the john-arthur team in task 4 of semeval 2023. the best submission of the team is ranked in position 3 of the leaderboard while the team itself is ranked 2nd. the core of the solution is an deberta-v2xxlarge fine-tuned model with some pre-processing and post-processing custom operations. the code of the best performing system of the team are opensourced with the hope that it can benefit the community foot_4 .