K now C omp at D ial AM -2024: Fine-tuning Pre-trained Language Models for Dialogical Argument Mining with Inference Anchoring Theory (2024)

Abstract: In this paper, we present our framework for DialAM-2024 TaskA: Identification of Propositional Relations and TaskB: Identification of Illocutionary Relations. The goal of task A is to detect argumentative relations between propositions in an argumentative dialogue. i.e., Inference, Conflict, Rephrase while task B aims to detect illocutionary relations between locutions and argumentative propositions in a dialogue. e.g., Asserting, Agreeing, Arguing, Disagreeing. Noticing the definition of the relations are strict and professional under the context of IAT framework, we meticulously curate prompts which not only incorporate formal definition of the relations, but also exhibit the subtle differences between them. The PTLMs are then fine-tuned on the human-designed prompts to enhance its discrimination capability in classifying different theoretical relations by learning from the human instruction and the ground truth samples. After extensive experiments, a fine-tuned DeBERTa-v3-base model exhibits the best performance among all PTLMs with an F1 score of 78.90% on Task B. It is worth noticing that our framework ranks #2 in the ILO - General official leaderboard.

## introduction dialogical argument mining is an emerging field that aims to bridge the gap between the analysis of argumentation and dialogue @xcite @xcite . traditional argument mining approaches have often focused on opinion mining within monological texts @xcite or document form contents @xcite @xcite . however, real-world argumentation frequently occurs in dialogical contexts, where multiple participants engage in a dy- apart from the dialogical information extraction paradigms explored by previous works @xcite , a generic modelling formalism for extracting dialogical information is the inference anchoring theory (iat) introduced by @xcite . it offers a systematic approach to decomposing text speech into distinct units (adus), while also anchoring and categorising logical inferences between propositions and locutions. as such, iat provides a comprehensive methodology for analyzing the maneuvers of dialogues within a given theoretical framework, thus building an explicit scaffolding for language models to handle semantics analysis tasks @xcite . based on this theory, dialam-2024 workshop @xcite introduces the first shared task in dialogical argument mining, aimed at modeling argumentation and dialogue information together within a domain-independent framework. the proposed tasks of dialam-2024 involves classification of the three-way argumentative relations between locutions and corresponding propositions, detection of relevant dialogical components and completion of the inference anchoring map. due to the in-context learning ability of llms on unconventional tasks with demonstrated examples @xcite , our initial attempt was to use large language models (llms) as the classifier for illocutionary relations @xcite @xcite . a combination of zero-shot and few-shot @xcite prompts integrated with chainof-thought @xcite were tested. however, we observed that popular llms, such as gpt3.5turbo (openai, 2023), fail to show significant understanding of the task and yield relatively low performance after exhaustive experiment. notably, recent developments in pre-trained language models (ptlms) on text classification tasks @xcite have empowered us to build our system the other way round. after the compilation of paired adus of propositions and locutions nodes embedded in a meticulously designed textual prompt, we fine-tuned our ptlms on the reconstituted dataset as that of a traditional text classification task @xcite @xcite . using this method, we were able to achieve relatively high accuracy in the identification of illocutionary relations. the classification results of task b were then used as textual information to assist the identification of propositional relations. an extensive ablation study was also conducted to test the effectiveness and generalizability of our proposed system. a maximal f1 score of 78.90% and precision of 82.35% on task b was achieved using a fine-tuned deberta-v3-base model @xcite . it is also noted that deberta-v3-large underperforms its base version, with a precision difference of -0.2%. the proposed explanation is that the model already converges on the given dataset, provided the base version parameters. several other ptlms, including roberta @xcite and albert @xcite are also trialed using identical prompt design, which yield results inferior to deberta-v3. however, fine-tuned ptlms converges inconsistently for task a, with a recall of only 33.79%. we suspect that besides text from adjacent propositions and locutions, the system will need more in-context information (e.g., a dialogue 2-3 nodes away) to assist the process of relation identification according to recent works on reasoning under contexts @xcite @xcite . as such, our proposed system provides valuable insight for dialogical argument mining using ptlms on a iat layout, and future works should be more focused on the revamp of methodology in in-context training information extraction. our code and results are publicaly available at arwenwutietie/dialam-2024 ## problem definition in this section, we would introduce the dataset format and elaborate on the formal definition of the shared task in dialam-2024. ## system overview in this section, we will introduce our proposed system. our method conducts sequential inferences where we predict y li 1 , y li 2 and s in the first stage, then infer y t s with the predicted s in the previous stage. ## experimental setup we followed a standard approach to partition our input data into training and validation sets. please refer to appendix c for more details. ## results and analysis in this section, we demonstrate our experiment results and conduct analysis on the issue we encountered through the experiments. our overall result is shown in further experiments indicate that the accuracy of s-node classification is greatly affected by the size of the training set. according to our observation, when 60% of the data is sampled for training, the accuracy on the test set reaches the highest (65.73%), and when all data is used for training, the accuracy decreases to 56.76%. we suspect that this may be due to model's overfitting to the training data. ## conclusion in this paper, we present our system for the dialam-2024 dialogical argument mining task, focusing on the identification of propositional and illocutionary relations within dialogues. by leveraging the iat framework, we developed a methodology that integrates human-defined prompts to stimulate ptlms' reasoning. our approach features commendable results in the identification of illocutionary relations with concise preprocessing procedures, as evidenced by our high f1 score and precision in task b. despite the notable success in task b, our system encountered challenges in task a, particularly in achieving consistent recall rates. this indicates that additional context beyond adjacent propositions and locutions may be necessary for enhancing the identification of argumentative relations. our findings contribute valuable insights into the application of ptlms in dialogical argument mining. the results underscore the importance of designing effective prompts and highlight the need for ongoing methodological advancements to fully harness the capabilities of ptlms in complex argumentation analysis tasks.