Overview of D ial AM -2024: Argument Mining in Natural Language Dialogues (2024)

Abstract: Argumentation is the process by which humans rationally elaborate their thoughts and opinions in written (e.g., essays) or spoken (e.g., debates) contexts. Argument Mining research, however, has been focused on either written argumentation or spoken argumentation but without considering any additional information, e.g., speech acts and intentions. In this paper, we present an overview of DialAM-2024, the first shared task in dialogical argument mining, where argumentative relations and speech illocutions are modelled together in a unified framework. The task was divided into two different sub-tasks: the identification of propositional relations and the identification of illocutionary relations. Six different teams explored different methodologies to leverage both sources of information to reconstruct argument maps containing the locutions uttered in the speeches and the argumentative propositions implicit in them. The best performing team achieved an F1-score of 67.05% in the overall evaluation of the reconstruction of complete argument maps, considering both sub-tasks included in the DialAM-2024 shared task.

## introduction argument mining @xcite investigates the automatic extraction of argument structures from natural language inputs. the nature of argumentation, however, can be very variable depending on its context, presenting significant differences between written and spoken argumentation @xcite , and between monological and dialogical argumentation @xcite . research in argument mining has mainly focused on the extraction of arguments only considering argument annotations such as premises and claims @xcite or attacks and supports between propositions @xcite @xcite , without bringing into consideration addi-tional relevant information that could be extracted from the speeches uttered in the dialogues. this is mostly due to the lack of presence of a framework annotating dialogue information in addition to argument structures in argument mining work. inference anchoring theory (iat) was proposed as an annotation framework for dialogue argumentation where not only the structure of arguments is captured, but also the speech acts and speaker intent is also annotated to support and contextualise argumentation in dialogues @xcite . therefore, when approaching argument mining in dialogues, iat represents an ideal framework to expand the standard annotations typically used in argument mining research (i.e., attacks and supports between premises and claims) integrating dialogical information to the argument mining process. although several corpora and resources annotated with iat such as us2016 @xcite , qt30 @xcite , rip @xcite , or fore-cast @xcite have been released in the last years foot_0 , there is a lack of systematic research in dialogical argument mining integrating these speech features into argument mining systems. dialam-2024 represents the first shared task in dialogical argument mining bringing together argument and speech annotations in an attempt to systematically explore the potential benefits of combining both when developing argument mining systems to be used in transcribed argumentative dialogues. the dialam-2024 shared task received submissions from six different teams exploring a broad set of approaches to integrate and combine argument and dialogue features for argument mining. in this paper, we describe the dialam-2024 shared task, summarise the most important aspects of the submitted systems, and provide an in-depth analysis of the final results of the shared task. furthermore, we perform a qualitative analysis of the output of the best performing system, pointing out the open challenges that will need to be addressed in future work. ## submissions fifteen different teams registered for the dialam-2024 shared task, of which six submitted their system outputs during the evaluation phase. all the submissions addressed the two sub-tasks of the shared task: (a) identification of propositional relations, and (b) identification of illocutionary relations. the submitted systems include a broad set of different language modelling approaches and model architectures. a summary of the submitted systems together with the references to their system description papers where the approaches are described in-depth is provided below. ## results in order to provide an insightful analysis of the performance of the submitted systems to the dialam-2024 shared task, we have divided the evaluation into three parts. first, the evaluation of the submitted systems when identifying propositional relations. second, the evaluation of the submitted systems when identifying illocutionary relations. finally, a global evaluation of the submitted systems when reconstructing argument maps looking at both, argument and discourse structures together. furthermore, each evaluation is also done considering two different setups: by considering exclusively the related pairs of nodes in the evaluation maps (i.e., focused), and by considering the complete map including non-related nodes (i.e., general). ## qualitative analysis to expand the findings observed in the analysis of the results based on the performance scores achieved by the participants, we carried out a qualitative analysis looking at the content of the submitted argument maps leading us to interesting observations. for that purpose, we compared the maps generated by the submitted systems with the eleven gold standard maps included in our test set focusing on specific aspects that influenced the performance of the systems including conflict relations or more complex argument structures such as convergent (i.e., figure 3 ), divergent (i.e., figure 2 ), and linked arguments (i.e., figure 3 ). in general, we observed that the submitted argument mining systems had problems recognising conflict relations, failing to identify most of them, and assigning conflicts between non-conflicting propositions. we also observed, in line with the previously reported results that, teams dfki-mlst and pungene were the ones that produced the most similar outputs compared to the gold standard maps. it was also interesting to observe how, the maps produced by team pokemon's system contained a significantly larger amount of relations compared to the rest. this is the reason of their higher scores in the focused evaluation with a significant drop of performance in the general setup. although both systems had some problems detecting more complex argument structures, we observed that dfki-mlst did a better job than pun-gene on these ones, identifying more convergent, divergent, and linked arguments. in the case of convergent and divergent arguments, the directionality of the relations is fundamental, making the identification of inference relations more challenging. for example, in the test map number 3, dfki-mlst identified correctly the inference relations, but failed to correctly place the one making the argument divergent. on the other hand, pungene had a rephrase instead of an inference, and the assigned directionality made the argument convergent instead of divergent, a similar error was observed in the test map number 4. in the test map number 5, dfki-mlst captures correctly the structure of the convergent argument, but fails to identify the two linked arguments. when looking at test map number 6, we observed again that both dfki-mlst and pungene had problems modelling the correct direction of the inference relations, identifying a convergent argument instead of the divergent one existing in the map, as seen within figures 2 and 4 . with respect to linked arguments, team dfki-mlst identified them in the test maps 7, 8, and 10, but represented them as convergent arguments instead of linked. a linked argument is represented by a unique inference relation including multiple premises rather than multiple inference relations between the premises and the claim, which would make it a convergent argument. we found, probably due to the implementation of the submitted systems, that this specific case was never considered as an output. in the test map number 10, we included a long linked argument, which consisted of six premises linked together in a unique inference relation towards the claim of the argument. although the linked relation was not correctly represented (it was modelled as a convergent argument instead), it is interesting that dfki-mlst correctly identified the six premises supporting the claim in this very particular case. finally, we also observed that in the evaluation set, none of the submitted systems was able to capture reported speech connecting locution nodes with illocutionary relations. ## conclusion this paper presents dialam-2024: the first shared task in dialogical argument mining. from the final results, we have been able to observe how the submitted systems that performed better in the dialam-2024 shared task either addressed both tasks at the same time (modelling argumentative and dialogical features altogether), or first focused on task b and then task a, showing that considering speech acts and dialogical structures helped to improve the performance in the overall reconstruction of argument maps. furthermore, from our qualitative analysis of the best submissions, we observed that there is still room for improvement in this area, specifically regarding the complex argument structures of convergent, divergent, and linked arguments, where not only the type of relation (i.e., inference) but also its directionality is of utmost important. it was also interesting to observe how, although the illocutionary relations were modelled with a reasonable success, specific cases such as reported speech represented a challenge for the systems submitted to the task. therefore, with the dialam-2024 shared task, it is possible to observe the complexity of argument mining from a new dimension, pointing future work towards a more complete modelling of argumentation, including illocutionary forces and complex argument structures.