S t FX - NLP at S em E val-2023 Task 4: Unsupervised and Supervised Approaches to Detecting Human Values in Arguments (2023)

Abstract: In this paper, we discuss our models applied to Task 4: Human Value Detection of SemEval 2023, which incorporated two different embedding techniques to interpret the data. Preliminary experiments were conducted to observe important word types. Subsequently, we explored an XGBoost model, an unsupervised learning model, and two Ensemble learning models were then explored. The best performing model, an ensemble model employing a soft voting technique, secured the 34th spot out of 39 teams, on a class imbalanced dataset. We explored the inclusion of different parts of the provided knowledge resource and found that considering only specific parts assisted our models.

## introduction semeval-2023 task 4: human value detection focuses on classifying a textual argument into one of 20 human value categories, each from schwartz' value continuum @xcite . examples of categories include, self-direction: thought (it is good to have one's own ideas and interests), face (it is good to maintain one's public image), and humility (it is good to recognize one's own insignificance in the larger scheme of things). blending computer science and social science, the system can also be applied to a variety of topics outside of academia. understanding the motives of others allows one to have more insightful conversations as they approach topics from different angles @xcite . by understanding what the other side of argument values, it can help us to reach common ground in discussions and lead to more productive interactions with less friction @xcite . we approached this task with models that consider both supervised and unsupervised techniques. furthermore, we perform an analysis on the data provided but the task to observe trends among different classes. the best performing model ranked 77 th out of the 111 submitted. the code is open source and available on github foot_0 . our unsupervised learning model employed a cosine similarity comparison between record embeddings and class description embeddings, taken from the knowledge resource foot_1 . to address some limitations of the system, the methods in which the data was preprocessed and embedded could have impacted the performance of our models. on top of data preprocessing, the dataset was only in english, limiting the reach of our models. ## background the submissions focused only on the valueeval'23 dataset @xcite , which contains almost 9, 000 records, in english. roughly 80% of the records came from ibm's argument quality dataset @xcite , 15% from the conference for the future of europe @xcite , and the remaining 5% from group discussion ideas. these arguments were then split by the task organizers into separate training (60%), validation (20%), and testing (20%) sets. the raw data was broken down into four columns; the id of the argument (to link the record to its labels), the conclusion, the stance, and the premise. the annotated label file has a record for every argument in the training and validation dataset, along with labels corresponding to the 20 classes. ## system overview the four end-to-end pipelines implemented for this task can be seen in figure 1 . each row pertains to a specific model, with arrows detailing the data flow for each model. nodes with identical names represent the same process occurring across different pipelines. figure 1 : the end-to-end pipeline for the system. ## experiments in this section, we discuss the design of our experiments and our hyperparameter tuning. our team name for this task was "jesus of nazareth". ## conclusion in working on this task, a wide range of machine learning models, both supervised and unsupervised, were implemented and tuned to maximize the primary metric used to evaluate models submitted, the f 1 score. supervised learning techniques performed best when employing "soft" voting and having a somewhat-shallow depth. when tuning the unsupervised learning model, using only phrases 1 and 3a outlined in figure 2 , taken from the knowledge resource, yielded the best results. other combinations of phrases led to unclear distinctions between classes, leading to the unsupervised model having a more difficult time predicting labels, and dragging down performance. the models were able to surpass the organizer's bert implementation's f 1 score on classes with lots of representation in the training data, but unable to surpass the bert model's overall f 1 score on the test data. if we were to revisit this project in the future, we would like to delve further into the unsupervised learning approach by interpreting the classes and data in different ways. when updating the supervised learning models, employing more of the data, for example, embedding the "conclusion" and "stance" columns, could be beneficial in more accurately predict the label of each premise. it would also be interesting to employ other stateof-the-art embedding methods and compare the results.