Friedrich Nietzsche at S em E val-2023 Task 4: Detection of Human Values from Text Using Machine Learning (2023)

Abstract: Literature permeates through almost every facet of our lives, whether through books, magazines, or internet articles. Moreover, every piece of written work contains ideas and opinions that we tend to relate to, accept or disregard, debate over, or enlighten ourselves with. However, the existence of subtle themes that are difficult to discern had inspired us to utilize four machine learning algorithms: Decision Trees, Random Forest, Logistic Regression, and Support Vec- tor Classifier (SVC) to aid in their detection. Trained on the ValueEval data set as a multi- label classification problem, the supervised ma- chine learning models did not perform as well as expected, with F1 metrics hovering from 0.0 to 0.04 for each value. Noting this, the lim- itations and weaknesses of our approach are discussed in our paper.

## introduction the creation of language remains one of humanity's greatest achievements, through which concepts, knowledge, stories, memories and emotions are encapsulated into words and conveyed across generations. among the multitude of ideas contained within the ever-growing expanse of written texts, whether it be news articles, novels, magazines or philosophical and religious scripture, there exists a commonality: the commentary on humanistic archetypes of freedom, creativity, selfpreservation, etc. upon which rational beings base their actions and thoughts on. these notions are expressed textually in varying degrees, sometimes apparent while at other times too subtle and difficult for the layman to recognize. for example, advertisements and political campaigns try to identify with potential customers and voters by openly appealing to their human values such as religious beliefs and concerns about nature and the environment. at other times, however, ambiguous and obscure literature could foster misunderstandings, misinterpretations, and even conflict. therefore, detecting and determining human values in texts could serve a potent role in many natural language processing applications, including the categorization of text information for tag-based recommendation systems, opinion/stance detection in articles, and analysis of complex literature written by poets and philosophers. as part of the semeval 2023 valueeval task @xcite , we seek to explore the effectiveness of supervised earning classifiers for semantic analysis through the utilization of algorithms like decision trees, logistic regression, support vector machines, and random forest in the extraction of implicit themes within the literature. with most research on semantic analysis being conducted using more powerful transformer/neural network based models, our paper chose to focus on more traditional machine learning techniques for their potential in being integrated with lightweight, portable recommendation systems and software based on thematic analysis of literature. due to its vast array of libraries for machine learning/data analysis/natural language processing applications, python was the primary language used for our study. with the problem being of multi-label nature and the semeval2023 dataset as our main subject of analysis, the data features underwent term frequency-inverse document frequency (tf-idf) vectorization before being fed to ensembles of each machine learning model for training in order to predict 20 human value themes on the test data. the hyper-parameters for each model were also tuned to inspect for changes in performance. ultimately, we observed that the aforementioned techniques did not generally perform as well as their deep learning counterparts, as reflected in our placing last in the rankings for scoring the main test set provided by semeval. the limitations of the tf-idf vectorization, coupled with the lower complexity of machine learning algorithms compared to neural networks, hindered the overall scoring performance of our ensembles. nevertheless, the paper's primary purpose is to serve as both a cautionary and foundational study for future research on this subject. the ethics data set @xcite ) is a new benchmark that can predict moral judgments. they also show to assess a language model's knowledge of morality. the creation and evaluation of these data sets is a significant step in setting up benchmarks for human emotion and they pave the way for training models and subsequently training intelligent agents guided by ethically sound directives. ## experimental setup due to the multi-label nature of the task and our data set comprising of both text and categorical features, feeding them into the models at once had given rise to various incompatibility issues that forced us to look for alternative approaches. the stages of our final decided approach will be explored in this section. ## results and analysis as shown in figure 3 , the supervised learning algorithms achieved considerably significant average f1-scores when trained and tested using a 70-30 split of the labelled valueeval dataset. selecting the support vector classifier as our competing model due to it obtaining slightly higher precision rates for the classes, we expected it to attain similar scores when predicting valueeval's evaluation test set. however, one of the biggest problems with our method was the tf-idf vectorization. due to it only considering word frequency and converting terms into sparse vectors of simple ones and zeroes, the technique fails at capturing context depending on the phrase or sentence provided. moreover, unlike unsupervised algorithms like glove and word2vec, tf-idf cannot extract patterns on its own and solely relies on a static factor like word count to generate word vectors. the simpler complexity of traditional machine learning models when put next to their deep learning counterparts cannot be ignored either. while bert-based architectures consist of multiple layers with each serving a purpose in the semantic recognition of text data, the svc we utilized is a more 'straightforward' method of predicting classes through margin distances. ## conclusion and future work finding human values in argumentative texts can be useful for a number of applications, including value-based argument generating, value-based personality profiles, and argument-faceted searches. a study of human values has the potential to expand prospective research in each of these applications. with this study being our first foray into natural language processing, we observed that multi-label human value prediction can be convoluted due to steep pre-requisites of data preprocessing and need of more complex machine learning models for better extraction of semantic relations between arguments and conclusions. despite the lackluster performance of our approach, we still believe in the potential of traditional machine learning algorithms to be a low-cost alternative to neural networks for classification of thematic literary values. as no field of research is ever truly conquered, we plan to further our research to include bert-based models, unsupervised vectorization methods, and more literary data sets for not only enlightening machines, but ourselves to witness the beauty of our writings.