On Classifying whether Two Texts are on the Same Side of an Argument (2021)

Abstract: To ease the difficulty of argument stance classification, the task of same side stance classification (S3C) has been proposed. In contrast to actual stance classification, which requires a substantial amount of domain knowledge to identify whether an argument is in favor or against a certain issue, it is argued that, for S3C, only argument similarity within stances needs to be learned to successfully solve the task. We evaluate several transformer-based approaches on the dataset of the recent S3C shared task, followed by an in-depth evaluation and error analysis of our model and the task’s hypothesis. We show that, although we achieve state-of-the-art results, our model fails to generalize both within as well as across topics and domains when adjusting the sampling strategy of the training and test set to a more adversarial scenario. Our evaluation shows that current state-of-the-art approaches cannot determine same side stance by considering only domain-independent linguistic similarity features, but appear to require domain knowledge and semantic inference, too.

## introduction same side stance classification (s3c) is the task to predict, for a given pair of arguments, whether both argue for the same stance @xcite . it abstracts from conventional stance classification, which, for an individual argument, predicts whether it argues for or against a corresponding issue. the hypothesis underlying s3c is that it can "probably be solved independently of a topic or a domain, so to speak, in a topic-agnostic fashion". 1 successful s3c can, for instance, help to quickly identify coherent posts in social media debates, or to quantify groups of posts with opposing stances. to advance s3c as a task in the argument mining community, this paper makes three main contributions: (1) development of new transformer-based approaches which improve upon the state of the art. (2) renewed assessment of the original s3c shared task dataset, and compilation of new training and test sets that enable a more realistic evaluation scenario. s3c shared task the s3c dataset @xcite is derived from the args.me corpus @xcite and comprises pairs of arguments from several thousand debates about one of two topics, namely abortion and gay marriage. the arguments have been retrieved from online debate portals. argument pairs were sampled from single arguments that occurred within the same debate context. binary labels for pairs were inferred according to whether or not the two arguments take the same stance. two tasks have been defined based on this data: within, where training and test sets contain pairs from both topics, and cross, where the training set is composed of arguments from the abortion topic, and the test set only contains gay marriagerelated argument pairs. ## experimental setup following the results of the shared task, transformer-based language models, such as bert, currently are the most successful approach to s3c. based on this previous work, we experiment with more recent transformers, carrying out the following three experiments. experiment 1: optimization we reproduce the shared task in its original form as well as the best-performing approach at the s3c shared task of @xcite . it serves as a baseline for comparison and represents the state of the art. the approach is based on bert with pre-trained weights for the english language. argument pairs are fed as a sequence pair into the model, and the pooled output of the last layer is used for binary classification. this architecture is fine-tuned with binary cross-entropy loss for three epochs, and a learning rate of 5e-5. in addition, we experiment with newer transformer-based pre-trained networks: roberta @xcite , which improved bert by using larger and cleaner datasets for pre-training; xlnet @xcite , which employs autoregressive pre-training; distilbert @xcite , which utilizes knowledge distillation during pre-training; and albert @xcite , which, among other things, uses embedding matrix compression and sentence order prediction as a pre-training task. for the within task, this procedure ensures nonoverlapping of pairs, there is a severe overlap of individual arguments between training and test. also, single debates from which pairs are sampled vary greatly in size. to test the influence of these two observations regarding overfitting effects, we first create an extended set containing all n(n -1)/2 possible argument pairs per debate, and then sample three new dataset splits of roughly comparable size, but with varying degrees of overlap of single arguments (cf. the two disjoint splits ensure that (almost) no single argument seen during training is reoccurring in a test set pair. this is achieved by splitting either across distinct debates (within), or across topics (cross). 3 the last split creates a test set which ensures that only one single argument from each pair is also contained in the training set. experiment 3: adversarial examples in the third experiment, we manually create an artificial test set @xcite to reveal the ability of our best model to solve different types of "adversarial" cases for same stance prediction more systematically. we select 25 distinct arguments from the "gay marriage" topic that are short and express their stance clearly. for each selected argument, we construct new arguments of four distinct types to obtain two pairs, one with same stance, and one with opposing stance. the first type, negation, is a simple negation of the argument. paraphrase alters important words from the argument to synonymous expressions with the same stance. the third type, argument, uses an argument from the same topic and stance, but semantically completely different regarding the first one. citation repeats or summarizes the first argument and then expresses agreement or rejection (a case frequently occurring in the dataset). the last three types are also formu-lated in a negated version to create additional test instances for the opposite stance. this results in a test set of 175 cases (see ## evaluation we report accuracy (a) and macro-f1 scores (f1) as experiment results. experiment 1: for the within task, we randomly split the official within training dataset into 90% for training and 10% for testing. for the cross task, we select all within pairs of the official training dataset assigned to the abortion topic as training data, and all gay marriage pairs for testing. for both tasks, another 10% of the sampled training sets are used as validation set during our experiments. this strategy creates an evaluation scenario equivalent to the official s3c shared task, but with slightly less training data. experiment 2: results for the second experiment in for the random split, we sample pairs from the entire set of all possible pairs per debate, 90 % for training and 10 % for testing. for the disjointwithin set of non-overlapping single arguments between training and test set, we utilize the information about debate origin from each argument pair. we split the dataset along distinct debate ids, so that we obtain (roughly) 90% for training and 10% for test. the disjoint-cross sets are split across the since the first two splitting strategies involve a random selection, we repeat the selection process five times and report average results. all tested scenarios surpass the majority baseline approving that the model actually learns to recognize (dis-)agreement of arguments. in accordance with the results from experiment 1, s3c works accurately (86.6% f1) for the randomly composed test set. however, for the two disjoint datasets with no overlap of individual arguments, the performance drops severely (ca. 62% f1). the performance for within does not even surpass the cross performance which is trained on a completely different topic. and in the single scenario, where one argument of a test pair has been seen during training, the performance is with 65% f1 rather low. experiment 3: a close inspection of misclassified pairs from the previous experiment of the disjoint test set reveals typical cases which require certain logical inference capabilities to obtain a correct same side stance prediction. based on this, we manually crafted the test set for the third experiment. for adversarial cases, even our best model only achieves 43.4% accuracy (41.7% f1-score). the confusion matrices in figure 1 show that the model is able to capture shallow semantic similarity between arguments (paraphrase) successfully. in contrast, it is not capable to predict the semantically more challenging types (argument and citation). negation, leading to opposing stance, is completely overlooked. ## discussion the experiments show that s3c performance drastically decreases for unseen arguments (experiment 2), and for difficult, adversarial cases (experiment 3), which undermines our confidence in the results from experiment 1. considering argument pairs composed of previously unseen individual arguments as a common scenario for s3c, the high performance on the official shared task dataset appears too optimistic. how can these differences between the original and our new scenarios be explained? let us recall: pairs of the original s3c dataset originated from single debates. it must be noted that debate size, i.e., the number of argument pairs sampled from a single debate, follows a power-law distribution (e.g., the experiment 1 training set contains 17,187 pairs from combinations of 251 arguments from the largest debate alone). fine-tuning of a transformer model now causes that re-occurring arguments of the same stance presented in different combinations get attracted to each other in the embedding space. arguments of opposing stance from one debate, in contrast, get repulsed. if enough combinations of argument pairs of one debate are presented to the network, embeddings of the pro and the con stance eventually form clearly separable clusters. this results in a task-specific overfitting on certain debates. each of the n unique arguments from one debate occur up to n -1 times in the training pairs. the model's performance thus correlates with the size of a debate when test pairs are sampled from the same debates as the training pairs. in fact, slicing the results from experiment 1 across different debate sizes reveals that test pairs originating from the five largest debates are predicted with nearly 100% accuracy. for smaller debates, the accuracy drops to the level of non-overlapping dataset splits. ## conclusion we carry out experiments to investigate the same side stance classification task. our results show that recent transformer models improve over the state of the art in the recent s3c shared task. with 73.7% f1-score, the best performance is achieved by the albertv2 model. we find, however, that the shared task's experimental setup suffers from overfitting, yielding overly optimistic results. a manually crafted test set of adversarial cases shows that all models fail on adversarial cases involving negation and citation of opposing arguments. from these results, three conclusions can be drawn for the improvement of the same side stance classification task: (1) for a more realistic evaluation scenario, training and test set pairs should be sampled from distinct sets of arguments. 4 (2) when the training set involves re-occurring arguments in different pairings, machine learning models should pay particular attention to measures against overfitting. for instance, a validation set should not be randomly sampled from the training set. (3) the hypothesis underlying the s3c task was that it can be solved in a topic-agnostic fashion. however, even our best model struggles to accurately predict the cross-topic scenario, or complex cases involving different arguments expressing the same stance. this finding suggests that the basic s3c hypothesis is not entirely true. for such cases, topic-specific knowledge and a deeper semantic representation of individual arguments than those encoded by current transformer models would be needed.