Bridging Argument Quality and Deliberative Quality Annotations with Adapters (2023)

Abstract: Assessing the quality of an argument is a complex, highly subjective task, influenced by heterogeneous factors (e.g., prior beliefs of the annotators, topic, domain, and application), and crucial for its impact in downstream tasks (e.g., argument retrieval or generation). Both the Argument Mining and the Social Science community have devoted plenty of attention to it, resulting in a wide variety of argument quality dimensions and a large number of annotated resources. This work aims at a better understanding of how the different aspects of argument quality relate to each other from a practical point of view. We employ adapter-fusion (Pfeiffer et al., 2021) as a multi-task learning framework which a) can improve the prediction of individual quality dimensions by injecting knowledge about related dimensions b) is efficient and modular and c) can serve as an analysis tool to investigate relations between different dimensions. We conduct experiments on 6 datasets and 20 quality dimensions. We find that the majority of the dimensions can be learned as a weighted combination of other quality aspects, and that for 8 dimensions adapter fusion improves quality prediction. Last, we show the benefits of this approach by improving the performance in an extrinsic, out-of-domain task: prediction of moderator interventions in a deliberative forum.

## introduction although people have been dealing with the art of persuasion since ancient times, there are many answers to the question of what constitutes a good argument or good argumentation, and none can be considered the best: the quality of arguments is complex, subjective and depends on the context in which the quality is assessed and on the prerequisites (attitudes and values) of the one who judges it. despite the high complexity of this task, a considerable research effort has been done to automatically model argument quality in different contexts @xcite due to its usefulness in downstream applications, such as automatic writing assistants @xcite , argument extraction @xcite and generation @xcite . social science offers another whole field of theories and definitions about argument quality in which the focus is usually not only on the argument itself but on the discussion between participants thus emphasizing the deliberative goal of the discourse @xcite . these two research communities, argument mining (am) and deliberative theory (dt), have not only produced different theories of argument quality, but also a number of annotated datasets on the basis of which the models for the automatic assessment can be (or have been) trained. in both am and dt, argument quality (aq) and its social science counterpart, deliberative quality (dq) are broken down into finer-grained dimensions. such dimensions map, for example, whether an argument is logically constructed (micro-level), or constructive in the context of an overall discussion (macro-level). however, neither individual quality dimensions, nor an aggregated score can do justice to the complexity of this concept. besides, a model that represents different aspects but has only been trained on one dataset will reproduce data-specific biases and may be less robust on other domains. multi-task learning (in this work, each quality dimension, e.g., logical cogency, clarity, persuasiveness, is a task), and the training data drawn from different datasets are a solution, as they allow to integrate the different dimensions and data sources from the two research communities. we propose to implement it using adapters @xcite , modules added between the layers of a transformer model. differently from fine-tuning of the full model, adapters allow to use a minimal amount of parameters while still achieving good performance. differently from standard multi-task learning, adapters do not require all the tasks to be learnt simultaneously, but they can be learnt as specific modules that can also be combined (fused or stacked). the modular design of the adapters then allows for flexible composition of the individual quality aspects and thus can be used in various configurations; this property facilitates future research of argument quality in different domains and lends itself as tool for the investigation of the relationship between different quality dimensions, both within and across disciplines. we experiment with 6 datasets containing aq and dq annotations, for a total of 20 dimensions (aq:8; dq: 12) covering a wide range of logical, rhetorical and dialectical aspects and a variety of domains and topics. our work proceeds in two steps. in the first step, we employ adapter fusion to learn a target dimension as a weighted combination of single-task adapters (e.g., clarity as a combination of cogency and effectiveness). we improve the results with respect to single-task learning on 8 dimensions out of 20 in a low-resource scenario. furthermore, fusion activation patterns provide us a tool to investigate the relationship between different quality dimensions. in the second step, we employ quality adapters for a new task on a new dataset: predicting that a post in a deliberative forum needs moderation @xcite . a fusion based on quality adapters is compared to baseline and full fine-tuning, outperforming both. moreover, our analysis shows that in solving the task, the models exploit information from all major quality sub-categories. crucially for the downstream application, casting moderation prediction as a fusion of quality adapters allows us to provide recommendation explained along specific quality dimensions (e.g., "this comment has major issues with the logical side of argumentation and it is disrespectful"). the contributions of this paper are at multiple level: a) at the level of task and methods, this is the first work which employs adapters for finer-grained aq dimensions; b) at the conceptual/theoretical level, we make a first step in the integration of theories of aq and dq, bridging between the annotations produced by the two communities and proposing adapter activation as a tool to empirically compare the conceptual core shared by aq and dq dimensions c) at the level of application, we show that quality adapters can support the task of predicting moderation of user comments, additionally contributing a theory-based explanation layer. ## related work argument quality much work on automatic modeling and annotation of argument quality (aq) in the argument mining community focuses either on a specific aspect of quality (e.g. argument relevance @xcite , sufficiency @xcite ) or a more general notion of argument quality based on human intuition @xcite . @xcite proposes a holistic taxonomy based on different theories of argument quality, inspired from rhetoric and linguistics, which divides aq into three main sub-categories. the logical dimension measures whether an argument has premises and a valid conclusion (cogency) thus takes the content and structure of a single argument into account. the rhetorical dimension (effectiveness) measures the persuasiveness of the argument and takes into account how it is presented (style, emotional appeal). the dialectical dimension (reasonableness) plays a more important role in the context of a discourse and reflects whether an argument is valid towards a universal audience (e.g. whether the reasoning is based on values generally accepted by the society) or whether it is constructive in helping to resolve issues. @xcite construct a corpus consisting of 302 arguments annotated with the three core and 15 sub-dimensions. wachsmuth and werner (2020) investigate which linguistic features are predictive of the fine-grained dimensions and which of the dimensions can be automatically assessed based on the textual input representations alone. the work by @xcite are the first that try to combine aq definitions from different corpora and based on different annotation schemas into one model. they investigate the generalizability of aq when combining different sources and explore multi-task learning for assessing it in four different datasets. on top of that they investigate the relationship between aq and other am tasks such as evidence detection. while most work on aq in the argument mining community focuses on the logical dimension or specific aspects of persuasion, research on deliberative quality (dq) from social science puts the discourse as a whole and the interaction between discourse participants into the focus. here, argument quality (or discourse / deliberative quality) is investigated to find out which tools and solutions (e.g. moderation, platform design, structured overviews) can contribute to a more productive and respectful public discourse. thus, the annotated datasets from this domain complement the ones from the am community providing many aspects of the rhetorical and dialectic dimensions. adapters adapters @xcite are a set of task-specific parameters that are introduced in every layer of a transformer @xcite and updated for a specific task while the rest of the pre-trained language-model parameters is kept frozen. besides being more efficient than full fine-tuning, adapters can be used as building blocks for other tasks due to their modular architecture and are therefore particularly well suited for transfer-and multi-task learning @xcite and to inject external knowledge sources to solve downstream tasks @xcite . @xcite propose to train task-specific adapters first (knowledge extraction) and combine them in a second step (knowledge composition) using selfattention to mitigate catastrophic interference, a problem which often occurs with traditional multitask learning approaches. in their work, this approach has proven to be useful especially in lowresource settings which is often the case for complex annotations such as the aq ones. to the best of our knowledge, our work is the first to employ adapters for aq to conduct a systematic comparison of aq and dq on different data sources. ## datasets for our experiments we rely on diversity, both in terms of data sets and different conceptualizations of argument quality. therefore, we also integrate two datasets from the social sciences, which are not established in the argument quality community, but show a particularly large variety of dimensions. europolis @xcite : consists of transcriptions of a face-to-face discussion about the topic immigration, initiated by the european union in order to enable deliberation on a european level. the spoken multi-lingual contributions have been transcribed, partially translated and annotated with five different dimensions of dq by political scientists, each dimension between two to five labels that can be arranged on a scale from a low to a high standard of deliberative abilities. the dimensions capture the logical aspect (justification), rhetorical aspects (storytelling) and dialectic aspects (common good, interactivity and respect). thf/bk @xcite : this dataset contains comments from two online citizen dialogues on munic-ipal issues: one on the further development of the "tempelhofer feld" site in berlin and the other on the use of the former lignite area in north rhine-westphalia. the data was annotated by political scientists with different dimensions of dq using a binary label for each dimension. the goal of the work was to investigate the relationship between "classic standards of deliberation", such as rationality and constructiveness and alternative forms of deliberation, such as humor, narratives and the use of emotions. this dataset therefore offers annotations for the so far rather underexplored and more affective dimensions of argument quality, such as positive emotions, narration and empathy. kialo @xcite : this dataset was created based on the online discussion platform kialo https://www.kialo.com on which users engage in structured discussions about a certain statement. users are able to rate the impact of an argument given its context. the dataset contains arguments about a large number of different topics together with their impact -a label which aggregates impact votes by all users. @xcite and @xcite report f-macro scores between 0.56 and 0.58 using different transformer-based models. grammarly argument quality corpus (gaq) @xcite : this dataset contains online contributions from four different domains annotated with the coarse-grained levels of the taxonomy introduced by @xcite on a five-point scale. @xcite evaluate different systems for automatic prediction of the quality scores, also experimenting with different multi-task architectures showing that multi-task learning can lead to improvements for all dimensions. ibm-rank-30k @xcite : the largest available corpus with aq annotations has been created based on a large quantity of binary annotations for human-generated arguments. the authors evaluate different methods of aggregating the annotations into a continuous score and conduct experiments on the automatic prediction of these scores with a pearson correlation of around 0.48 on a test set with unseen topics. @xcite found positive correlations between this aggregated aq score and automatically generated scores for cogency, effectiveness and reasonableness on this corpus. swanrank @xcite : as one of the first datasets with aq annotations in the am community this corpus contains arguments from on ## experiment 1: modeling aq and dq using adapters and adapter-fusion in the following experiment we are interested in the relationship between different conceptualizations of aq and dq from a modeling perspective: does injecting knowledge about other dimensions help to improve the predictions on a target dimension? if so, which dimensions are especially helpful? to investigate this we treat each of the 20 dimensions as a task which we aim to model. we want to compare how the models perform without external information (using only a single-task adapter) with those using information about other dimensions (using multi-task learning with adapter-fusion). ## results can we improve modeling aq with adapterfusion? ## analysis: relationship between aq and dq dimensions for each target dimension we analyze which adapters get activated during inference. we extract the attention scores for source dimensions for each target dimension based on the test set. similar to @xcite we assume that high activations indicate more useful source tasks. ## experiment 2: predicting moderator interventions in this experiment, we evaluate the models using a new down-stream application: we want to predict whether a comment in an online discussion should be moderated. our hypothesis is that we can use information about different quality dimensions to solve the task. moderation, especially in deliberative discussions such as on civic participation platforms, is a complex task that generally consists of facilitating a productive and fair discussion with respectful interaction. since the task becomes more difficult for human moderators to perform as the number of participants and comments increases, automatic models can be useful for predicting whether a comment should be moderated. here, the logical quality dimensions can help distinguish less argumentative from argumentative comments, the rhetorical dimensions are important for ensuring civil interaction, and the dialectical dimensions can identify valuable comments (is a solution proposed or the common good considered?). we use the dataset from @xcite , in which the authors annotated the functions of moderation in discussions on a deliberative platform and identified 'quality of comments' as a common reason for intervention. the dataset was used in @xcite , who obtained an f1 score of 0.34 using a full fine-tuning approach with roberta-base. the dataset is small and consists of 876 negative and 222 positive instances, a further motivation for a multi-task based approach. we train and test the models on the 5-fold split provided by @xcite . as moderator interventions are the minority class we use class weights for all models. we compare the following models: (quality) scores st: we generate predictions for each quality dimension and convert them into scores. 3 . classifier: logistic regression. (quality) scores-mt: similar to quality scores but we generated with the fusion-based adapters. classifier: logistic regression. moderation-st: we train a single-task adapter on the task of moderation intervention. fusion-aq: we train a fusion on the task of moderation intervention using only quality adapters as input representations. fusion-all: we train a fusion on the task of moderation intervention using all quality adapters and the adapter for moderation. roberta-full: we report the result of can aq adapters be applied to predicting moderator interventions? ## conclusion this work targeted the relationships between different aspects of argument and deliberative quality. we experimented with 6 datasets and 20 quality dimensions, employing adapters we learn modular representations of the targeted dimensions. we show that adapter fusion improves predictions in 8 dimensions out of 20. we then use the learnt adapters in the task of predicting moderator interventions -we show that information about different argument quality dimensions helps to improve the performance. having more insights about which aspects of argument quality were activated more or less when the model that a user contribution should be moderated could help human moderators decide which aspects to focus on and information about why that model fired can increase human moderators awareness of and (ideally) confidence in automated support methods. we make models for single-task adapters and fusions and the code to train and test them available: https://github.com/blubberli/ argqualityadapters.git .