DFKI - MLST at D ial AM -2024 Shared Task: System Description (2024)

Abstract: This paper presents the dfki-mlst submission for the DialAM shared task (Ruiz-Dolz et al., 2024) on identification of argumentative and illocutionary relations in dialogue. Our model achieves best results in the global setting: 48.25 F1 at the focused level when looking only at the related arguments/locutions and 67.05 F1 at the general level when evaluating the complete argument maps. We describe our implementation of the data pre-processing, relation encoding and classification, evaluating 11 different base models and performing experiments with, e.g., node text combination and data augmentation. Our source code is publicly available.

## introduction dialam 2024 @xcite is the first shared task in dialogue argument mining. it uses the inference anchoring theory (iat) framework @xcite as data schema. iat describes argument structure as graphs of propositions that are derived from the argumentative discourse units (adus; the basic units of argumentative analysis). the shared task focuses on the detection and classification of the relations that (1) argumentatively link these propositions with each other (ari) and that (2) anchor them in the corresponding adus (ilo). the dialam dataset is based on the qt30 corpus @xcite , which is a collection of 30 episodes of the show question time by the bbc. the dataset includes transcriptions of dialogues between a moderator and several panelists and audience members annotated according to the iat. figure 1 visualizes the structure of the data. in simplified terms, iat models argumentation information as a bipartite graph. 2 one side consists of the adus as they occur in the original text, called locution (l) nodes, and transition (ta) nodes that link them in the direction of the dialog flow. the other side consists of information (i) nodes which encode the propositions derived from the locutions and the argumentative relation (s) nodes (e.g., inference, conflict, or rephrase) that connect them in the direction of argumentative reference. finally, i and s nodes are anchored by illocutionary relation (ya) nodes in l and ta nodes, respectively, i.e. they encode from which l and ta nodes they are derived. the relation nodes connect to their arguments via two distinct roles: incoming (edges point towards the relation node) and outgoing (edges point away from it). the shared task data are organized in nodesets where each nodeset is a collection of annotated nodes and edges in argument interchange format @xcite extracted from an episode. dialam poses some unique challenges because it requires three different types of relations to be extracted (see figure 1 ): argumentative relations between propositions (s nodes; subtask 1), illocutionary relations modeling speech acts (ya:l-to-i nodes; subtask 2.1) and, relations between argumentative relations and dialogue turn transitions (ya:ta-to-s nodes; subtask 2.2). note that all relations have at least one incoming and outgoing edge, but argumentative relations (s nodes) such as inference may have more than one incoming edge. thus, subtask 1 is an instance of n-ary relation extraction. furthermore, ya: ta-to-s relations link ta and s nodes which are both relation nodes, so this is a meta-relation. both aspects circumvent usual relation extraction approaches that assume binary relations connecting spans over text. there are 25 relation labels in total with a very imbalanced distribution (see appendix a and e.2). previous approaches to dialogue argument mining, such as @xcite , have shown that transformer-based models work well on the argument relation identification task, with roberta @xcite significantly outperforming bert @xcite , xlnet @xcite , distilbert @xcite and albert @xcite . they have found that in many cases misclassified relations were due to the lack of context or multiple valid interpretations of a relation. however, @xcite address a simpler task compared to the dialam setup because they classify only propositional relations while dialam involves n-ary relations between different types of nodes (propositions, locutions and meta-relations). our contributions are as follows: (1) we introduce a unified approach towards dialogue argument mining based on n-ary relation classification and train a single model that can handle all three types of relations to get the most out of the data, (2) our dfki-mlst submission achieves the best scores in the global setting of the shared task, and (3) we conduct a comparative analysis of different types of base models, explore data augmentation, weighted loss and node text combination. ## system architecture we handle all three subtasks by framing them as n-ary relation classification. let r @xmath0, a)|@xmath1, @xmath2, j, l)} of labeled spans over a text t with i and j start and end indices with respect to t and l the label. we define n-ary relation classification as assigning a class in the following we explain how we construct relations r st , i.e. sets of argument-role -text-span pairs over a single text, and the relation classes c from the individual relation nodes. in detail, we describe how we normalize the data (2.1), encode each task data as n-ary relations (2.2), and, finally, how we implement the relation classification (2.3). ## experiments and results with the relation classification approach described in section 2 we train our model on the dialam data. we split the original training set into training (1259 nodesets) and validation (140 nodesets) partitions and repeat the training procedure three times with different seeds. the best model is selected based on the validation set performance. our dfki-mlst submission uses deberta-v3 foot_3 @xcite as text encoder trained with a learning rate 1e-4 and a window size of 512 tokens foot_4 for 20 epochs on a single gpu nvidia h100 80gb hbm3. we evaluate our model with the official script that outputs precision, recall and f1 scores for the ari and ilo tasks, and the global metrics represent the combined scores. all scores are calculated at two levels: focused (only related arguments/locutions) and general (complete argument maps). ## conclusion this paper introduces the dfki-mlst submission that achieves the best scores in the global evaluation setting of the dialam shared task. we describe our nodeset pre-processing pipeline and the system architecture. we also present the comparison of different base models (deberta, bart, mistral etc.) as well as our experiments with data augmentation, class distribution and node text combination. we observe that some models (e.g., deberta) demonstrate better performance on the argument relation task while other models (e.g., bart) are better at detecting illocutionary relations.