How to Handle Different Types of Out-of-Distribution Scenarios in Computational Argumentation? A Comprehensive and Fine-Grained Field Study (2024)

Abstract: The advent of pre-trained Language Models (LMs) has markedly advanced natural language processing, but their efficacy in out-of-distribution (OOD) scenarios remains a significant challenge. Computational argumentation (CA), modeling human argumentation processes, is a field notably impacted by these challenges because complex annotation schemes and high annotation costs naturally lead to resources barely covering the multiplicity of available text sources and topics. Due to this data scarcity, generalization to data from uncovered covariant distributions is a common challenge for CA tasks like stance detection or argument classification. This work systematically assesses LMs’ capabilities for such OOD scenarios. While previous work targets specific OOD types like topic shifts or OOD uniformly, we address three prevalent OOD scenarios in CA: topic shift, domain shift, and language shift. Our findings challenge the previously asserted general superiority of in-context learning (ICL) for OOD. We find that the efficacy of such learning paradigms varies with the type of OOD. Specifically, while ICL excels for domain shifts, prompt-based fine-tuning surpasses for topic shifts. To sum up, we navigate the heterogeneity of OOD scenarios in CA and empirically underscore the potential of base-sized LMs in overcoming these challenges.

## introduction argumentation as a communication tool for human reasoning has engaged researchers over millennia @xcite @xcite figure 1 : common ood types of computational argumentation covering evidence classification @xcite , and mono or multilingual stance detection @xcite . and has become an important research area in natural language processing under the umbrella of computational argumentation @xcite . specifically, computational argumentation (ca) models human argumentative processes and leads to complex tasks such as stance detection @xcite and argument quality evaluation @xcite . however, developing resources for such ca tasks requires significant annotation efforts @xcite , which often inadequately capture the wide range of heterogeneity in available text sources and topics. this situation makes ood scenarios, especially those involving significant covariant distribution shifts, a common challenge for ca tasks since lms are anticipated to generalize across such shifts in current and future applications @xcite . these shifts occur when input data distribution changes between the training and testing phrases and can be viewed as a specific aspect of out-of-distribution (ood) scenarios @xcite . this work focuses on three types of covariant distribution shifts frequently encountered in ca tasks: topic shift, domain shift, and language shift. figure 1 illustrates these three types of ood scenarios, in which researchers aimed at developing systems for ca tasks that generalize across unseen topics @xcite ), text domains @xcite , or languages @xcite . these studies have observed that ca systems often fail to handle ood scenarios. given this variety of ood scenarios in ca and the need for data efficiency, this paper aims to answer the following research question: "how to handle different types of ood scenarios in computational argumentation using lms"? most previous work on evaluating the generalization and robustness of nlp models has either predominantly focused on a single type of ood scenario, such as domain shift @xcite , or on general ood that does not distinguish among various types of ood scenarios @xcite . however, these studies overlook the heterogeneous nature of ood and, thereby, limit the transferability of the corresponding findings to the spectrum of shift types in ca tasks. this study introduces a detailed evaluation framework encompassing a list of holistic performance measures ( § 3) to pinpoint crucial generalization flaws such as misalignment between performance and training loss in models. in addition, we feature a heterogeneous collection of eleven ca tasks ( § 4) covering three types of ood scenarios. we evaluate these tasks with an extensive experimental setup ( § 5) covering twelve lms of various sizes and eight learning paradigms, including gradient-based learning like vanilla fine-tuning (ft) and prompt-based finetuning (p+ft), as well as in-context learning (icl). from the observed results ( § 6), we conduct an in-depth analysis ( § 7) to gain a better understanding of how learning paradigms and lms differ for different types of ood for computational argumentation. in contrast to @xcite , suggesting in-context learning (icl) surpasses fine-tuning lms for addressing general ood, we find different learning paradigms excel in different types of ood for ca tasks. in particular, icl outperforms gradient-based learning for domain shifts where train and test label distributions heavily differ. however, gradient-based learning surpasses icl for topic shifts characterized by a clear semantic divergence in the covered topics between the training and testing datasets. in summary, our main contributions are: out-of-distribution generalization studies in nlp target ood generalization from different perspectives, focusing on the robustness of lms @xcite @xcite or ood detection @xcite . similar to computer vision @xcite , nlp studies primarily focus on considering covariant distribution shifts @xcite and analyze single types of them in isolation, such as domain across datasets @xcite @xcite , language @xcite , topic @xcite . this shortage of comprehensively analyzing ood hinders analytical or methodological advancements in a challenging field such as computational argumentation since generalization of methods is limited when relying on shift-specific features @xcite @xcite . prompt-based fine-tuning commonly, pretrained lms are fine-tuned by providing a natural language input and optimizing regarding an arbitrary label @xcite . instead, promptbased fine-tuning @xcite ) (p+ft) allows relying upon acquired competencies during pretraining, both for encoding the input and predicting the label by formulating the task as a cloze test. this procedure allows lms to reach comparable performance to their large-sized counterparts @xcite with the same limited data as in few-shot settings. despite their success for few-shot scenarios, little work analyzed how p+ft generalizes differently than ft or how it performs considering complete datasets, exceptionally @xcite showed the robustness of p+ft against adversarial attacks. with this work (figure 2 ), we address the need to comprehensively evaluate ood abilities of lms with a particular focus on computational argumentation. in particular, we assess a variety of lms using in-context and gradient-based learning paradigms, considering three types of ood scenarios covering eleven different tasks. ## ca tasks across ood types in this section, we present the selection of computational argumentation tasks ( § 4.1) and subsequently show their heterogeneous distribution shifts, focusing on covariant and label properties ( § 4.2). ## experimental setup this section outlines the experimental setup covering the models, learning paradigms, and evaluation. ## results this section reports results on a detailed ( ## analysis next, we discuss four aspects which differentiate learning paradigms based on arg-cls where we observe prominent differences. figure 6 visualizes the layer-wise embeddings of the classification proxy tokens -cls for ft and mask for p+ft. it shows that p+ft retains more semantic information (about topics) until the last layers, while ft eliminates them across all layers during training. hinting, again, at substantial differences between ft and p+ft. ## conclusion this work marks the most extensive study to date addressing the heterogeneous types of ood scenarios in ca by systematically evaluating different ood types. we evaluate a multiplicity of lms and learning paradigms on eleven ca tasks. with this extensive evaluation, we shed light on the challenges of having diverse covariant distribution shifts in ca. in addition, we provide takeaways of general relevance, such as the superiority of icl for domain shifts, where gradient-based learning fails to generalize effectively due to significant label discrepancies between the training and testing data. in contrast, gradient-based learning surpasses icl when generalization across significant semantic differences is required, like in cases of topic shifts. with the rise of larger lms, systematic evaluation of distribution shifts becomes even more important, necessitating the consideration of additional factors such as computational efficiency, task complexity, and data contamination. finally, our findings highlight the untapped potential of base-sized models, which points towards a need for further advancements in gradient-based learning paradigms.