Quintilian at S em E val-2023 Task 4: Grouped BERT for Multi-Label Classification (2023)

Abstract: In this paper, we initially discuss about the ValueEval task and the challenges involved in multi-label classification tasks. We tried to approach this task using Natural Language Inference and proposed a Grouped-BERT architecture which leverages commonality between the classes for a multi-label classification tasks.

## introduction everyone has a perspective on how they approach a problem and make a decision, from the simplest decisions such as choosing to lend a pen to a friend to decisions that affect their life and everyone around them. these decisions are made consciously based on some of the values that they strongly believe in. the task 4 of semeval-2023 -valueeval @xcite , is to understand which of these human values form the basis for someone's decision in a textual argument. valueeval @xcite presents the task of understanding human values as a natural language processing problem, by proposing an annotated dataset of textual arguments, with the labels being the human values that would be drawn to make that decision. each argument in this dataset is provided with a premise, conclusion and stance. lets consider this example argument from the dataset with premise "marriage is the ultimate commitment to someone, if people want it they should be allowed", the conclusion "we should abandon marriage" and the stance "against". the task is to figure out why someone takes a particular stance (in favor of/ against) for the conclusion, given the premise -"what human values led to someone taking this particular stance?" the task studies the (often implicit) human values behind natural language arguments, such as to 1 https://github.com/ajaymopidevi/grouped-bert have freedom of thought or to be broadminded. values are commonly accepted answers to why some options are desirable in the ethical sense and are thus essential both in real-world argumentation and theoretical argumentation frameworks. however, their large variety has been a major obstacle to modeling them in argument mining. in their dataset, mirzakhmedova et al. (2023) also provided each argument with a set of human values in the form of labels that are closely aligned with psychological research. for each argument, these 20 labels provide greater detail about which human values are inferred for making the decision. ## background this classification task becomes much more challenging as it is a multi-label classification problem. in classification problems, deep learning architectures try to update their layer weights to emphasize the output of softmax layer of the correct class by making it closer to 1, while also making the outputs of the other classes closer to 0. in a multi-label classification problem, that's not possible as each input may have multiple labels. tsoumakas and katakis (2007) provides a list of approaches used to tackle multi-label classification problems and provides a comparison of the performance of these approaches. they showed that pt3 transformation @xcite provides better results compared to other transformations. if an input belongs to both the classes a and b, then pt3 transformation generates a new class c, such that input has belonged to only class c (which represents both a & b). considering that each argument has 20 labels, a pt3 transformation generates many more classes, and also reduces the number of samples per class, making the data very sparse. in such tasks with more labels, pt4 transformation @xcite is preferred as it uses l binary classifiers, with each binary classifier predicting 1/0 for each class. label cardinality(lc) is computed as the average of the true labels in the input data. label density(ld) is similar to label cardinality, but is also divided by the total number of labels in the input data (venkatesan and er, 2014). we use this information to draw further insights about our dataset. these values help us identify how sparse the labels are for each argument in our dataset. for the training dataset, we have observed the lc and ld values in ## system overview we model this problem as a natural language inference (nli) model, to predict the human values that led to the stance being taken based on the premise and the conclusion. nli typically has only two inputs i.e premise and conclusion and the output is stance. we modified our conclusion to have the stance followed by the actual conclusion. from here, the use of the term conclusion includes the stance appended to the beginning of the original conclusion. the premise and this new conclusion pair are jointly encoded using bert to obtain the contextual information between the premise and the conclusion along with the contextual information of the premise tokens and the contextual infor-mation of the conclusion tokens. this contextual information in the form of embeddings are transferred to the classification network. in this paper, we propose a grouped-bert architecture to handle different multi-label classification challenges. as each argument can belong to multiple classes, we tried to group some of the classes that often have the same representation. this grouping can generalize some of the core reasoning behind the arguments, that is common in all its output classes. instead of approaching a random grouping, we group a subset of labels in each category based on the similarities. for finding these similarities, we extract the embeddings of the classes and perform a k-means clustering. we used a constrained k-means algorithm proposed by @xcite to exactly model each group with 4 classes. we combined 4 classes in a group, as our label cardinality, i.e average number of labels per argument is around 3.4 the last hidden state of the pre-trained bert model is initially fed into group classifier, to identify the group of classes it belongs to. then the output of each group classifier is forwarded to the hidden layers and subsequently the l binary classifiers for each group class, and also to a separate classifier layer to predict whether the final set of labels belongs to that group. we need to create extra labels for these groups for loss calculation, by evaluating its member labels in that group. ## experimental setup for the input embeddings, we used embeddings generated from pre-trained 'bert-base-uncased' model. we trained our models for 200 epochs with an adamw optimizer with a variable learning rate, linearly rising to peak(1e -4 ) at 40 epochs and then linearly declining back to 0 by the 200th epoch. for the loss function, we considered a local loss(sum of crossentropyloss for each group) and a group loss(crossentropyloss for the group predictions) and the group is weighted 4 times that of local loss. we have grouped the classes as follows: the feed-forward network layer which takes as input the cls token from the bert output is a linear layer with 512 output channels. each of the group classifiers shown in figure 1 have 64 output channels. the outputs from this hidden layer of 5 group classifiers are then fed into separate networks, each having a single linear layer with 32 output channels. these outputs are finally fed into classifiers, each of which predicts one of the 20 classes. we trained the model on a laptop with an nvidia gtx 1070 gpu and intel 7700hq cpu. ## results for the main test dataset, our groupedbert achieved f 1 -score of 0.38, which is less than the baseline bert model (0.40). by grouping the classes, we expected that the groups would be independent i.e each argument can have only the values from that group. however, each argument in the training set still belongs to multiple groups, with the label cardinality being 2.54, which is not a much improvement from 3.4. this added more constraints to the original set of labels, rather than identifying the core representation in the grouped labels. the f 1 -scores by our model for the classes stimulation, hedonism, face, conformity:interpersonal, humility is 0. each of these value categories have the lowest distribution in their respective groups. we suspect that this uneven distribution of labels could be one reason why our model couldn't predict any of these categories correctly. as we were only able to train our model on 5 year old hardware, we were limited in our ability to train bert layers and instead had to rely on pre-trained bert layers with the weights frozen. this could be another contributing factor because of which we could not achieve a better performance with this model. ## conclusion we would like to test our groupedbert with a part of the dataset, having even distribution of all the classes and verify the label cardinality stays close to 1. these approaches might provides insights whether groupedbert architecture can be used for multi-label classification tasks.