Perturbations and Subpopulations for Testing Robustness in Token-Based Argument Unit Recognition (2022)

Abstract: Argument Unit Recognition and Classification aims at identifying argument units from text and classifying them as pro or against. One of the design choices that need to be made when developing systems for this task is what the unit of classification should be: segments of tokens or full sentences. Previous research suggests that fine-tuning language models on the token-level yields more robust results for classifying sentences compared to training on sentences directly. We reproduce the study that originally made this claim and further investigate what exactly token-based systems learned better compared to sentence-based ones. We develop systematic tests for analysing the behavioural differences between the token-based and the sentence-based system. Our results show that token-based models are generally more robust than sentence-based models both on manually perturbed examples and on specific subpopulations of the data.

## introduction identifying argumentation units is difficult, both for humans and machines. the challenge starts with the question of what it means for a segment to be argumentative towards a given topic in the first place @xcite . @xcite propose a pragmatic approach for defining arguments and ask annotators to identify segments that can be placed in @xmath0: "<@xmath1, @xmath2. they compare models that are trained to label tokens as being part of argumentative segments to models that classify full sentences as containing argumentative segments (arg) or not (non-arg) (see figure 1 ), ultimately arguing that token-based training is preferable. their experiments suggest that a token-based approach is more robust when sentence boundaries are unknown or not precisely given. argumentative segments provide reasons for taking a positive (pro) or negative (against) stance on a topic. these arguments are highly topic-specific, but the decent accuracy of cross-topic models indicates that there are also topic-independent cues. @xcite previously showed that transformer-based models learn to map specific cue words to a label and learn little about argumentation reasoning. it could be that this is the most we can expect in a cross-topic scenario. the question then remains where these cues are found: are they in the arg segments themselves or are they also provided by the non-arg context? when comparing token-based and sentence-based models, we expect token-based models to be better at picking up cues that are specific to arg segments themselves, whereas sentence-based models may be more susceptible to cues from the non-arg context, in particular, when these appear to announce an argumentation (e.g. because i think that...). reliance on (non-arg) cues is a particularly strong signal that general cues rather than reasoning are used. in this paper, we dive further into this line of research. we rerun experiments with the best models of @xcite to ensure a fair basis of comparison, reproducing most of the original results and coming close for the rest. we then design multiple robustness tests comparing the behavior of token-and sentence-based models in mixedsegment sentences, i.e. sentences that contain at least one arg segment and one non-arg segment. we expect token-based models to be more robust because they are trained to distinguish between arg and non-arg segments within sentence boundaries and thus have access to more precise information as to what makes up an arg segment during training. this hypothesis is confirmed by our perturbation tests, which also show different behavior on subpopulations of the data. we thus show that a relatively small, curated dataset of ad-figure 1 : an example sentence from the aurc-8 @xcite ) topic gun control. each sentence in the dataset has one vector of token-wise gold labels (in-and output in a sequence labeling approach, i.e. token-based) as well as one sentence-wise gold label (in-and output in a sequence classification approach, i.e. sentence-based). arg and non-arg gold segments are sequences of tokens that carry the same label. ## background and related work in this section, we first present related work on argument unit recognition ( §2.1) and then dive further into the concept of robustness tests ( §2.2). ## reproduction experiments this section describes our reproduction study, including the dataset used ( §3.1), the experimental setup ( §3.2), the model evaluation metrics ( §3.3), and the requirements for a successful reproduction together with our results ( §3.4). ## conclusion in this study, we partially reproduced the results of @xcite and introduced new robustness tests that showed how token-based models are generally more robust than models trained at a sentence level on an argument unit recognition task. we applied two type of tests: perturbations and subpopulations. with regards to the perturbations, we found that 1) removing the non-arg segment from a mixed-segment sentence, and 2) replacing the arg segment with a non-arg segment in announcing phrases such as 'their main argument @xmath3. we did not find a difference in bias among the two types of models with regards to semantic similarity between training and evaluation data, and high argumentative token ratios at the sentence level. instead, we showed that the development of perturbation test sets itself can shed light on specific subpopulations of the data: our token-based t1 t2 t3 model before after ∆acc before after ∆acc before after ∆acc token-based bertlarge .875 (.018) .832 (.033) -.043 -4.8% .760 (.035) .830 (.037) +.070 +9.2% .760 (.022) .683 (.037) -.077 -10.1% token-based bertlarge+crf .878 (.021) .856 (.024) -.022 -2.5% .790 (.015) .760 (.028) -.030 -3.8% .740 (.036) .672 (.023) -.068 -9.2% sentence-based bertlarge .835 (.034) .808 (.053) -.027 -3.2% .638 (.059) .640 (.064) +.002 +0.3% .652 (.047) .576 (.034) -.076 -11.7%