Diversity Over Size: On the Effect of Sample and Topic Sizes for Topic-Dependent Argument Mining Datasets (2024)

Abstract: Topic-Dependent Argument Mining (TDAM), that is extracting and classifying argument components for a specific topic from large document sources, is an inherently difficult task for machine learning models and humans alike, as large TDAM datasets are rare and recognition of argument components requires expert knowledge. The task becomes even more difficult if it also involves stance detection of retrieved arguments. In this work, we investigate the effect of TDAM dataset composition in few- and zero-shot settings. Our findings show that, while fine-tuning is mandatory to achieve acceptable model performance, using carefully composed training samples and reducing the training sample size by up to almost 90% can still yield 95% of the maximum performance. This gain is consistent across three TDAM tasks on three different datasets. We also publish a new dataset and code for future benchmarking.

## introduction topic-dependent argument mining (tdam) is the task of extracting argument components in documents or document collections @xcite . @xcite refer to it, information-seeking argument mining) means that argument components are directed towards a given topic @xcite . the topic is used in two ways: by a machine learning model to learn topic-relevance and as a query to retrieve input documents for automatic argument search @xcite . while llms (large language models) show astounding results @xcite , task-related datasets are still important to improve model performance @xcite @xcite and decrease certain undesirable behaviours @xcite via finetuning, and to provide curated data for evaluation purposes. to assemble large amounts of training samples, it is common to use non-experts to annotate datasets. however, in contrast to a task like sentiment analysis, the task of identifying arguments is not naturally understood by non-experts and, due to pitfalls like commonly used fallacies @xcite , needs a thorough training phase and strict quality control of the crowdsourcing process. hence, crowdsourcing datasets for tdam is not only time-consuming but also expensive, as it requires a large number of workers per sample for satisfactory agreement. for instance, @xcite report a sum of @xmath02,323 to only $290 (see appendix a.1) for a dataset created in the composition proposed in this work (see section 6.1). although showing difficulties on small samples sizes, we can still reduce the sample size by 67% with flan-t5 xl. we observe the same trend on the other two datasets: on the ibm-corpus, 15% of the training data with ernie 2.0 and 26% with flan-t5 xl are sufficient to reach 95% of the maximum performance; on the iam-corpus, 15% with ernie 2.0 and 30% with flan-t5 xl are sufficient. this clearly challenges the trend to develop larger datasets for tdam. following our proposed dataset composition makes low-budget productions of high-quality tdam datasets possible, contributing to a more diverse landscape of those datasets. generalization to other tasks: while we test our approach on tdam only, work on related tasks also shows performance improvements when focusing on dataset diversity. for instance, @xcite show that higher performance can be reached if subsets of stance detection datasets are sampled for diversity with an unsupervised topic model and used for training. sultan et al. (2020) use a transformer-based question generator and conclude that more diverse questions lead to a higher performance on downstream question-answering tasks. @xcite also show improvements for question-answering by generating questionanswer pairs with a focus on diversity conditions like spacial aspects, question types and entities. similar to our findings, they also show that performance gains are highest in low-resource scenarios. hence, we speculate that a well-architected dataset composition, based on diversity, can also lead to a high performance on question-answering and pure stance detection datasets. we publish our newly created dataset 1 and code 2 , allowing for further benchmarking experiments to develop the design of future tdam datasets.