A Robustness Evaluation Framework for Argument Mining (2022)

Abstract: Standard practice for evaluating the performance of machine learning models for argument mining is to report different metrics such as accuracy or F1. However, little is usually known about the model’s stability and consistency when deployed in real-world settings. In this paper, we propose a robustness evaluation framework to guide the design of rigorous argument mining models. As part of the framework, we introduce several novel robustness tests tailored specifically to argument mining tasks. Additionally, we integrate existing robustness tests designed for other natural language processing tasks and re-purpose them for argument mining. Finally, we illustrate the utility of our framework on two widely used argument mining corpora, UKP topic-sentences and IBM Debater Evidence Sentence. We argue that our framework should be used in conjunction with standard performance evaluation techniques as a measure of model stability.

## introduction deep learning models have obtained state-of-theart results on a wide range of natural language processing (nlp) tasks and have even achieved superhuman performance on benchmark tasks @xcite . the standard approach for evaluating machine learning models is to use held-out data and report various performance metrics such as accuracy and f 1 . however, reporting an aggregate statistic on benchmarks does not reflect the model's performance and robustness when applied to real-world texts. indeed, recent works have shown that nlp models are not robust to perturbations. for instance, natural language inference (nli) models classify a permuted example where word positions are randomly changed, as they would classify the original input @xcite , and sentiment analysis models give a lower sentiment score when a positive phrase is added to the original example @xcite . @xcite argue for rigorous evaluation to avoid poor generalisability, whereas @xcite propose systematic development of test suites. several frameworks have been developed for evaluating the robustness of nlp models, for example checklist @xcite , textattack @xcite , robustness gym @xcite , and textflint @xcite . there is limited work on evaluating the robustness of argument mining models @xcite , and the linguistic and logical reasoning required in argument mining tasks have so far been ignored. in this paper we propose a robustness evaluation framework for machine learning-based argument mining models. in particular, we propose a variety of simulation functions that, given a seed dataset, automatically create simulated datasets. the simulated datasets are designed to mimic realistic settings which can be used to test the model's robustness. our framework is model-agnostic and only requires access to the data. we propose several novel robustness tests tailored to the argument mining task (e.g. argument removal, motion syntax inversion, motion negation, motion synonym/antonym verb replacement, etc.) as well as re-purpose robustness tests previously applied to other nlp tasks (e.g. contract/expand contraction, verb tense change, back-translation, etc.). we focus on two major corpora available for argument mining: the ukp topic-based sentential argument mining corpus @xcite where the task is to determine whether a sentence is an argument for a topic and whether it supports or opposes the topic, and the ibm debater evidence sentences corpus @xcite where the task is to determine whether a sentence includes evidence for a given motion. while other works on robustness focus on adversarial training (e.g. morris et al. ( 2020 )), our contributions are a range of functions that generate simulated datasets that reflect real-world examples. we believe our robustness evaluation framework can be used to enhance the standard performance evaluation in order to create better models for argument mining. figure 1 gives an overview of our proposed robustness evaluation framework. ## related work there is a plethora of work in evaluating the robustness of nlp models that cover a variety of tasks: sentiment analysis @xcite @xcite @xcite @xcite , machine translation @xcite @xcite , natural language inference @xcite @xcite @xcite @xcite , question answering @xcite @xcite , duplicate question detection @xcite , and fake news classification @xcite . robustness is evaluated by perturbing data and checking whether the model responds correctly to these changes. amongst the most commonly used transformations (note that we use "perturbation" and "transformation" interchangeably in this paper) we find: punctuation errors @xcite , typos @xcite @xcite , synonym replacement @xcite @xcite @xcite , contractions @xcite , verb tense change @xcite , entity replacement (ribeiro et al., 2020), back-translation @xcite , negation @xcite @xcite , and using bert @xcite for word replacement @xcite . in this paper, we draw from previous works and apply commonly used data transformations in nlp tasks to argument mining. regarding task-specific perturbations, textflint includes perturbations for nli, machine translation, and sentiment analysis amongst others, while @xcite extend checklist with templates tailored for the nli task to cover more linguistic and logical reasoning such as causal, spatial, and pragmatic. to the best of our knowledge, only two works have considered the robustness of argument mining models, for topic-dependent argument classification models @xcite and stance detection @xcite . @xcite used simple linguistic transformations such as two typos and negation by adding the tautology "and false is not true" after each sentence. @xcite proposed more transformations such as punctuation errors, entity replacement, replacing a noun with its hyponym, using topic alternatives (e.g. death @xmath0, and adding speculative adverbs in the evidence text (e.g. cannabis leads to other @xmath1, and used these transformations in adversarial training. in both works, the sentence-level topic information within an argument or motion, which we believe to be a key aspect in argument mining, is ignored. in this paper, we propose a robustness evaluation framework and introduce a variety of novel transformations tailored for the argument mining task as well as use existing transformations for nlp tasks and apply them to argument mining. ## robustness tests for argument mining we first introduce the terminology used in this paper. given an original dataset with n instances @xmath2, x 2 , ..., x n }, where x i is a pair of texts, and a corresponding set of n labels @xmath3, y 2 , ..., y n }, we train a model f : @xmath4. we define a simulation function sim to be a function that takes a labelled dataset, called seed dataset, and creates a new, labelled simulated dataset s with the corresponding set of labels y ′ . for example, we may have (s, y ′ ) = sim(x , y), but other sub-sets of x could be used, such as the training set or the validation set. a robustness test consists of applying a simulation function to obtain a simulated dataset and then evaluating a model's robustness on the simulated dataset (see figure 1 for an overview). the model robustness is recorded as the difference between the model's performance on the original dataset and the model's performance on the simulated dataset. next, we describe the two argument mining datasets we use as seed datasets and the simulation functions we propose for obtaining simulation datasets that can be used to test the robustness of argument mining models. ## experiments in this section, we apply our proposed simulation functions and evaluate the robustness of argument mining models. we use ukp and ibm, respectively, as seed datasets. we adopt the methodology in @xcite and apply each simulation function on the original dataset to generate the corresponding simulated dataset. depending on the simulation function used, the simulated dataset may be of different size compared to the seed dataset. for example, some functions are not applicable to all instances in the seed dataset (e.g. contraction), while other functions may result in creating one example (e.g. punctuation error) or several examples for each instance in the dataset (e.g. synonym replacement, entity replacement). we experiment with bert @xcite , a pre-trained transformer network @xcite which set state-of-the-art performance on various sentence classification and sentence-pair classification tasks. we use bert-base-cased and fine-tune on each dataset. for ukp, we train using the proposed train-test-validation sets and we obtain 71.7% accuracy and 67.4% macro f 1 , using e-3 as learning rate and training for 21 epochs. for ibm, we split the dataset into 70% for training, 15% for testing and 15% for validation, and obtain 83.4% accuracy and 71% f 1 , using 2e-5 as simulation function original text in seed dataset new text in simulated dataset topic change (abortion, abortion is wrong because it is taking a human life.) (minimum wage, abortion is wrong because it is taking a human life.) argument removal abortion is wrong because it is taking a human life. abortion is wrong. motion topic synonym we should ban alternative medicine we should ban naturopathy motion syntax inversion private universities should be banned we should ban private universities motion negation we should subsidize cultivation of tobacco we should not subsidize cultivation of tobacco motion adverbial modifier we should ban lotteries we should absolutely ban lotteries motion adverbial modifier we should further exploit wind turbines we should indeed exploit wind turbines motion syn verb replacement we should abolish the monarchy we should get rid of the monarchy motion ant verb replacement we should prohibit flag burning we should permit flag burning motion similar replacement we should fight global warming tattoos should be banned motion different replacement we should fight global warming we should subsidize renewable energy punctuation (single) the war on poverty has not had any effect in the 40 + years that it has been going on. the war on poverty has not had any effect in the 40 + years that it has been going on punctuation (all) it is true, as conservative commentators often point out, that some minimum-wage workers are middle-class teenagers or secondary earners in fairly well-off households. it is true as conservative commentators often point out that some minimumwage workers are middleclass teenagers or secondary earners in fairly welloff households typo milton friedman called them a form of discrimination against low-skilled workers. milton friedman calledt hem a form of discriminatio nagainst lwo-skilled workers. ## conclusion we proposed a robustness evaluation framework for machine learning-based argument mining models. our framework is model-agnostic and only requires access to the data. we presented 15 simulation functions, amongst which 6 are novel and tailored for the argument classification task by exploiting sentence-level topic information within an argument or motion, with the rest of the functions re-purposed for argument mining tasks. these can be used to automatically create simulated datasets, designed to mimic realistic settings which can be used to test the model's robustness. we illustrated the utility of our framework on two widely used argument mining corpora, ukp topic-sentences and ibm debater evidence sentence and showed that, while robust, bert models can still be vulnerable to new inputs. our robustness evaluation framework can be used to enhance the standard performance evaluation in order to create better models for argument mining by measuring model stability. we experimented with the major corpora available for argument mining, however our framework can be applied to datasets for relation prediction in argument mining @xcite . there are several avenues for future work. first, we plan to apply our framework to other datasets and models used in argument mining. we also plan to use the simulated datasets in adversarial training to evaluate whether model robustness can be improved. further, it would be useful to explore combining several simulation functions to create simulated datasets. finally, one interesting line of research is to provide explanations and/or summaries of failures on the simulated datasets that can be used to understand why a model fails and thus work on improving it.