Active Learning for Argument Strength Estimation (2021)

Abstract: High-quality arguments are an essential part of decision-making. Automatically predicting the quality of an argument is a complex task that recently got much attention in argument mining. However, the annotation effort for this task is exceptionally high. Therefore, we test uncertainty-based active learning (AL) methods on two popular argument-strength data sets to estimate whether sample-efficient learning can be enabled. Our extensive empirical evaluation shows that uncertainty-based acquisition functions can not surpass the accuracy reached with the random acquisition on these data sets.

## introduction argumentative quality plays a significant role in different domains of social activity where information and idea exchange are essential, such as the public domain and the scientific world. theoretical discussions about what constitutes a good argument can be traced back to the ancient greeks @xcite . researchers nowadays continue exploring this topic, trying out approaches that employ empirical machine learning estimation techniques @xcite . one of the most expensive and time-consuming tasks for machine learning-driven argument strength prediction is data labeling. here, the result is highly dependent on the quality of labels, while the annotation task demands cognitive and reasoning abilities. one way to guarantee good annotations is to perform labeling with schooled experts, raising project costs extensively. for this reason, a common approach involves employing crowd workers. as argument strength detection is a highly subjective task, crowd workers' labeling results are often identified by low reliability and prompt researchers to counter-check the results with more crowd workers and as specifically developed agreement-based techniques. sometimes a threshold for agreement cannot be reached at all, which might lead to data loss (see e.g. @xcite . this motivates us to investigate the applicability of some existing methods for reducing the amount of training data for automatic argument strength prediction. to this end, we look closely at the technique of active learning (al). in this paper, we evaluate standard uncertainty-based acquisition functions for the argument strength prediction. we perform several experiments for the task of binary argument-pair classification (see ## data set for our analysis, we use two publicly available data sets suitable for the task of pairwise argument strength prediction: because supporting and opposing arguments often share the same vocabulary and semantics, we do not treat each stance within a given topic as a separate topic, contrary to the authors of the two data sets. instead, we combine the "for" and "against" arguments within the same topic under the same topic index and, thus, avoid leakage of semantic information between train and test data split. this preprocessing makes the performance of our models not directly comparable with the performance from the original papers. however, reproducibility of the original papers' results is beyond the scope of this work, as our focus lies on testing al acquisition functions instead of reaching higher performance with our models. due to the high computational costs of the al process, we decide to select the three most representative topics from each data set. one way to reach high representativeness would be to select topics that are average in difficulty. since we try to approximate a real-world setting where the labels are unknown, it is not clear at the beginning which topics are more challenging to learn than the others. for this reason, we decide to select our test topics according to their size. thus, we cross-validate our models on each data set's smallest topic, the largest one, and the median-sized one. thus, the topics we select according to this procedure are topics 10 ("is the school uniform a good or bad idea?"), 13 ("tv is better than books") and 14 ("personal pursuit or advancing the common good?") in ukpcon-varg1strict and topics 3 ("does social media bring more harm than good?"), 4 ("should we adopt cryptocurrency?") and 7 ("should we ban fossil fuels?") in ibm-9.1kpairs data. ## results for the estimation of the performance of the models trained on the whole data with k-fold crossvalidation, we reach a comparable performance of our bert-based binary classification technique on both of the data sets (average accuracy on ukp-convarg1strict: 0.76, on ibm-9.1kpairs: 0.77). this is a slightly worse performance than @xcite achieves with the same architecture; the reason could be attributed to a different topic attribution strategy, as well as to some differences in the used hardware or hyperparameters, such as batch size or the number of epochs. we find that the topics selected by us from the ukpconvarg1strict stand rather on the low end of difficulty, with model accuracy tending towards the upper end of the scale when validated on these topics: all of them are higher than the mean performance of 0.76 (see appendix a.3 for more details). however, from the distribution point of view, two of the topics, namely 10 and 13, yield median model performance, making them, in our opinion, suitable representatives of the whole data. as for the ibm-9.1kpairs data set, our selected topics produce on average comparable performance with the model performance on the whole topic set (accuracy of 0.776 vs. 0.77 respectively). they also represent the most difficult topic, the easiest topic, and one closely neighboring the median topic (accuracy of 0.78 being slightly higher than the median performance of 0.77). in this case, the selected topics provide a better representation of the whole data set and grant strong validity when it comes to generalizing the results of our experiments. the series of experiments we conducted in order to test whether our proposed heuristics for al data acquisition provide us with any significant improvement surprisingly do not reveal any heuristic which would perform better than in the case of a random acquisition. this is true both for ukp-convarg1strict and ibm-9.1kpairs data; a detailed overview is presented in tables 2 and 3 . statistical significance of the results has been tested with a wilcoxon signed-rank test, which provides a nonparametric alternative to the paired t-test and is more suitable due to the non-gaussian distribution of the differences in the results. all heuristics result in performance that is lower than that of the random baseline. all of our results are statistically significant with p-@xmath0.0001. despite the fact that random acquisition turns out to be the best one in terms of performance, with our results being consistent through both data sets and the difference being statistically significant, it is still noticeable that the differences in each case are rather small (see figures 1 and 2 for graphic visualization of the model performance during active learning rounds comparing the acquisition functions). ## discussion the results of our experiments do not point to any acquisition functions which outperform random acquisition. this finding does not exclude the possible existence of some other suitable acquisition functions, even from the same class (such as uncertainty-based). this remains an open question and should be considered in further research on the topic. for the time being, the random acquisition should be considered the approach of choice when selecting data for labeling for the task of pairwise argument strength prediction. this is sensible both from an accuracy standpoint as well as due to the computational cheapness of a random process. as the literature suggests, a possible reason why uncertainty-based methods perform so unimpressively is their proneness to picking outliers -a disadvantage that some other methods, such figure 2: overview of the training results on the ibm-9.1kpairs dataset based on different uncertainty-based acquisition methods as diversity-based acquisition (e.g., @xcite , do not have. this might be especially critical in the realm of argument strength prediction, as outliers might represent the arguments where relative argument strength difference is marginal, the data are noisy, or where the provided labeling is too subjective. another critical factor is the cold-start problem, i.e., overfitting on the small initial data set of data, for which no initial informativeness estimation could be performed. this poses a drawback for the uncertainty-based methods, relying on the initial data sample for subsequent data acquisition. ## conclusion this paper evaluates the effect of uncertainty-based acquisition functions, such as variation ratios, entropy, and bald, on the model performance in the realm of argument strength prediction. as no acquisition function tested helps improve model performance in comparison to the random acquisition, we have not found any justification for using uncertainty-based active learning for pairwise argument strength estimation.