T eam EC at S em E val-2023 Task 4: Transformers vs. Low-Resource Dictionaries, Expert Dictionary vs. Learned Dictionary (2023)

Abstract: This paper describes the system we used to participate in the shared task, as well as additional experiments beyond the scope of the shared task, but using its data. Our primary goal is to compare the effectiveness of transformers model compared to low-resource dictionaries. Secondly, we compare the difference in performance of a learned dictionary and of a dictionary designed by experts in the field of values. Our findings surprisingly show that transformers perform on par with a dictionary containing less than 1k words, when evaluated with 19 fine-grained categories, and only outperform a dictionary-based approach in a coarse setting with 10 categories. Interestingly, the expert dictionary has a precision on par with the learned one, while its recall is clearly lower, potentially an indication of overfitting of topics to values in the shared task’s dataset. Our findings should be of interest to both the NLP and Value scientific communities on the use of automated approaches for value classification

## introduction and background this paper describes our participating system to the valueeval shared task (semeval 2023 task 4), which can be described as follows : "given a textual argument and a human value category, classify whether or not the argument draws on that category." @xcite . the dataset for this task was built from existing datasets with the aim to cover a wide range of cultural sensitives, while english is the only language considered @xcite . automated detection of human values is seen by the authors as an important tool in a broader policy analysis toolbox. indeed, recent research demonstrated that the design and acceptance of public policies is influenced by their underlying values and the way they are framed in values terms @xcite . our aim in participating in this task was to further explore the possible interplay between dictionary-based techniques -which have policyrelated application (e.g. @xcite learned dictionary approaches based on keyword extraction @xcite and more state-of-theart methods involving large language models @xcite . our contribution consists of: by doing so, we aim to stimulate debate and understanding on the respective merits of low/green tech approaches compared to state-of-the-art approaches based on large transformers models for complex classification tasks. ## systems and experimental setup we used two approaches: a dictionary-based and a transformer-based. the only training data used was the new data created specifically for this shared task by the organisers. we use an 80/20 train/dev split of the training data, and used the same splits for both approaches. it has to be noted that we evaluated only the dictionary-based approach before the official end of the shared task, and that the transformer-based model was evaluated on the same dataset but outside the official competition. for the dictionary-based approach, we preprocessed the text by lemmatizing it using nltk part-of-speech tagger and lemmatizer, and removed the stop words. we computed for each label the distribution over tokens, by splitting over white spaces, and filtering tokens with less than 3 occurrences in a class and removing all non-alphabetical tokens. then, taking inspiration from tf-idf we ranked the words: for each word, we took the inverse document frequency, considering each labelled text snippet as a document, and its tf-idf value for each class by considering the count for that particular class. we then selected for each class the n most important words as measured by their tf-idf weight. using this dictionary, in order to perform predictions, we pre-processed the text to be predicted in the same way we pre-processed the training set, then we computed the score of each class, where we count 1 for each matched word. because there was no easy way to set a threshold and that usually 10+ classes get predicted, we decided to rank the classes by their highest tf-idf score, and to return the top 4. we observed that in the dataset, the average number of labels was 3.42, which suggests that a fixed number of classes close to it would heuristically optimise the quality. experiments showed that 4 labels would optimise the score over the dev set. the best results were achieved by using the combination of conclusion and premise and not only the premise. we submitted two runs: for @xmath0, that the performance increase with respect to 400 was negligible and we thought it would be better to take a lower value in order to avoid overfitting. these dictionaries had respectively a total of 679 and 2430 words. in another run, we also used the same setting, but split the tokens around stop words instead of spaces, a rake-based approach @xcite , which shared about 75% of the vocabulary with the dictionary-based ones, the rest being multi-word expressions. for the transformer-based approach, we used the base model xlm-roberta-base @xcite that we fine tuned using the following hyperparameters: 1e -5 learning rate, batch size 16 and early stopping with a patience of 3 and using micro f1 score as the optimisation criterion. experiments using the train-dev set showed that both the f1 and the loss started to rise after 1500 optimisation steps. as such, we made a first submission using a model trained over 1500 steps over the full training dataset. the performance was below expectations, even lower than the dictionary method performance, therefore we submitted a new run by increasing the number of optimisations by a shift of 1000 steps until the model performance started to decrease on the test set. the optimal number of training steps was 4500 -surprisingly high considering our choice based on the dev set, and indicative of a possible severe overfitting of the model to the very specific data of the shared task. in terms of hyperparameter optimisation, we only tried increasing the number of steps, in an attempt to understand the unexpected behaviour of the transformer. ## experimental results we report in the first striking result is that the performance of the roberta classifier is not particularly high, reaching only 45.8 micro f1 at best, when overfitting the data of this shared task. this performance places it slightly above the median performance of runs submitted to the shared task, as there were a total of 111 official submissions. also surprising was that the optimal number of training steps determined on the train-dev splits used, yielded performance lower than the one of dictionary based approaches. the third surprise was, that the transformer-based approach has a f1 of only 6.1 points above the dictionary-based approach which uses only 100 words per class, as similar previous experiences in a 3-class settings yielded a difference of about 20 points @xcite in terms of micro f1. in the dictionary-based approaches have performances ranked a bit above the 3rd quartile of the shared task's leaderboard, indicating that clearly underperforming. nevertheless, the performance gap with our transformers system is not that significant, and it can even report better recall (for @xmath1. interestingly, a smaller number of words yields a very low precision but a good recall of 0.51, probably competitive with the best system, whose f1 score is 0.56. on top of that, as seen in when increasing the number of words, we observe a slight increase of 5 points in precision at the cost of an important decrease of almost 10 points in recall. for the transformer-based approach, the opposite behavior is observed when augmenting the number of training steps: with 1500 steps the precision is at its highest of 0.51, while with 4500 steps there is a drop of about 5 points in precision but 13 points increase in recall. lastly, we decided to compare the performance of the expert made dictionary from @xcite and the learned dictionary. the expert dictionary is defined on a coarser 10-class version of the taxonomy. we therefore made the evaluation by integrating the more fine grained 19 values into the original 10 values (face and humility were respectively mapped onto power and conformity). ## conclusion we took advantage of this shared task to answer two questions. firstly, how well do transformer-based solution perform compared to simple dictionary approaches in the values domain? secondly, how well does an expert designed dictionary perform vs. a learned one? we surprisingly find out that the transformers and dictionaries have the same level of performance on that specific dataset for a fine grained classification task, while performing better, but not drastically better, only in a coarse grained setting. we find that an expert dictionary has a similar precision, albeit lower recall than learned ones. these findings hint at a potential bias of topics toward specific values in the dataset, and call for further study when these models are applied to real world data. we believe this work provides valuable insights both to the nlp and the value scientific communities on the use of automated approaches for value classification.