Tenzin-Gyatso at S em E val-2023 Task 4: Identifying Human Values behind Arguments Using D e BERT a (2023)

Abstract: Identifying human values behind arguments isa complex task which requires understandingof premise, stance and conclusion together. Wepropose a method that uses a pre-trained lan-guage model, DeBERTa, to tokenize and con-catenate the text before feeding it into a fullyconnected neural network. We also show thatleveraging the hierarchy in values improves theperformance by .14 F1 score.

## introduction humans often come to different conclusions given the same premise. this variation can be attributed to their values. identifying the values behind the arguments is helpful in understanding the argument itself. downstream tasks like supporting or opposing argument generation can benefit from value identification. in this task @xcite , we aim to identify 20 value categories in a given premise, stance and conclusion pair. data used is collected from four geographical cultures, manually annotated by @xcite . this paper proposes a method for multi-label classification of the premise, stance, and conclusion pairs using encoder only lms as a background model. we use deberta @xcite , a pretrained language model, that has shown remarkable success in various nlp tasks, including classification. the proposed method tokenizes the premise, stance and conclusion text using the pretrained tokenizer, and then concatenates them and feeds it into the lm, generates a representation of the combined text, and maps it to a set of values using a fully connected neural network. the model is trained on a multi-margin loss function and evaluated on metrics such as accuracy, precision, recall, and f1 score. this approach can potentially result in a highly accurate and effective nlp model for identification of values in arguments. our major finding from the task is that the hierarchy in the values helps with identifying them. we found that adding 5 high level values self-direction, power, security, conformity, benevolence, universalism to the existing 20 values improved performance substantially compared to just using the 20. these 5 values are extracted from labels of 20 values. for example, the value self-direction: action belongs to the coarse self-direction class. our model outperformed the 1-baseline and random baseline in @xcite and performed similarly to the bert baseline. ## background the task involves predicting what values would have lead humans to come to the given stance and conclusion when provided with a premise. there are a total of 20 such values that needs to be identi- transformer layer 1 cls premise sep stance sep conclusion transformer last layer feed forward neural network values fied in an argument(premise, stance and conclusion pair). we are also provided with higher level values which are less nuanced than the required level to be predicted. figure 1 shows the various levels of values present in the data, that are associated with arguments. we are interested in the level 2 values in this task. following is an example of a premise, stance and conclusion along with values associated with the argument. stance is binary variable with 'in favor of' and 'against' as possible values. level 1 values: have freedom of action level 2 values: self-direction: action level 3 values: openness to change level 4a values: personal focus level 4b values: growth, anxiety-free ## system overview we used an end to end neural approach to predict the values in an argument. transformer language models have shown remarkable capabilities in nlp applications. we make use of deberta model which is an encoder only model pretrained on data from wikipedia (english wikipedia), bookcorpus @xcite , openwebtext @xcite , and stories (a subset of common-crawl @xcite ). joint representations of premise, stance and conclusion are generated using deberta. this is done by concatenating the premise, stance and conclusion with sep token text transformer layer 1 transformer layer n-2 transformer layer n-1 transformer layer n classifier classifier classifier classifier classifier classifier l1 values l2 values extracted values l3 values l4a values l4b values separating them. concatenated text is then tokenized and fed into lm and last hidden state of cls token is used as the representation of argument. this facilitates information exchange between premise,stance and conclusion while generating representation of argument. this representation then used as input to a feed forward neural network with dropout and silu activation function. ffnn maps the representation to 20 logits which corresponds to the values. we optimize multi margin soft loss to train the model end to end to perform multi-label classification. following are some of the variations of this approach we tried. ## experimental setup we used the main dataset provided in the task website. it consists of a train split with 5393, a validation split with 1896 samples and a test split with 1576 samples. level 2 values has 20 classes with high imbalance. the maximum class count to minimum class count ratio in train split is 12 and overall is approximately 10. this ratio for level 1 is 84, level 3 is 2, indicating that there is very high imbalance in levels that high number of values. levels 4a and 4b has almost equal distribution of labels. level 3,4a and 4b values are absent for 2024 out of 5393(37%) arguments in training split. we fill the missing values with 0,1 values from an uniform random distribution. models mentioned in 3 are optimized using adam optimizer with weight decay and learning rate 1e-4. deberta model from huggingface @xcite is used and end to end model is trained with batches of size 16 without any gradient accumulation. we use precision, recall and f1 score as metrics to evaluate the models as they are resistant to class imbalance problem when averaged using macro strategy. scikit-learn's @xcite implementation of macro measures are used.we use performance of models on val split to compare the models. ## results ## conclusion in this paper we discuss the task to identify values in a given premise, stance, and conclusion pair, which can be useful in downstream tasks like generating arguments. we found that identifying the hierarchy in the values improves performance, and adding 5 high-level values to the existing 20 values significantly improved the model's accuracy compared to just using 20. hierarchical methods did not perform as expected due to missing high level values. the proposed approach has the potential to be an effective nlp model for identifying values in arguments.