Pungene at D ial AM -2024: Identification of Propositional and Illocutionary Relations (2024)

Abstract: In this paper we tackle the shared task DialAM-2024 aiming to annotate dialogue based on the inference anchoring theory (IAT). The task can be split into two parts, identification of propositional relations and identification of illocutionary relations. We propose a pipelined system made up of three parts: (1) locutionary-propositions relation detection, (2) propositional relations detection, and (3) illocutionary relations identification. We fine-tune models independently for each step, and combine at the end for the final system. Our proposed system ranks second overall compared to other participants in the shared task, scoring an average f1-score on both sub-parts of 63.7.

## introduction this paper is a system design paper for the dialam-2024 task. this task involves the creation of dialogue annotations from dialogue text. specifically, annotations in the format of a graph under the inference anchoring theory (iat) framework. the iat @xcite framework allows for dialogue argumentation annotations in a way that retains relevant information and structural data irrespective of domain. for this task, we are provided with a dataset that contains numerous .json files where each document represents a graph under the iat framework. the data used is the qt30 corpus @xcite , where dialogue is taken from 30 episodes of the show question time. our system is a pipeline that splits the tasks into three steps. at the first step we utilize bertscore to produce similarity scores to find connections. then, for each step, we fine-tune a bert model to perform multiclass classification using information gained from previous steps as input. we finetune each model separately and combine it into a pipeline at the end where we create a finished graph. with regards to scoring, we were second in figure 1 : example final output. the blue nodes on the left are i-nodes (propositions), the blue nodes on the right are l-nodes (locutions), the orange nodes are snodes (relations), the yellow nodes are ya-nodes, and the purple nodes are ta-nodes (which depict a transition from one utterance to the next). the ordering of the pipeline mentioned later is also depicted starting with green to yellow then to red as the final step. the general case, beating third place by 8 points. we also scored third in the focused setting. ## method to tackle the problem of populating this graph, we decided to isolate each part to create a pipeline. rather than following the subtasks laid out, we split it into three different steps instead. the first step is to connect the i-and l-nodes and label the ya-nodes that lie between them. secondly, as we now know the ordering of the i-nodes, we can then label the s-nodes that go between each i-node. thirdly, we would connect the ta-nodes to the created s-nodes and label the ya-nodes that go between them. this ordering is labeled in figure 1 as well. ## results and analysis the main method used to measure the success of our system is by calculating precision, recall, and macro-f1. a score will be calculated for subtask a (ari) and subtask b (ilo) each as well as a global score which is the aggregate of the two. furthermore, they will be split into two different versions: focused and general. focused evaluates the performance looking at only the related classes in the evaluation files only while general also includes the non-related class. this means that a high performance in the general version but low performance in the focused versions shows a pessimistic approach (overly relies on the non-related class) while the inverse shows an optimistic approach that relates too many propositions and locutions. another thing to note is that a big downside of our pipelined system is that it is very prone to cascading errors. this is also an additional reason as to why in step 2 we opted to use as few inputs as possible in order to prevent the cascading of errors. the only part which used a lot of the information from the previous steps was step 3. however, the added information allowed the model to get an f1 score of 96.2 which is very strong. from ## conclusion and future work overall our system performed quite well, especially on subtask b which was the identification of illocutionary relations. the recurring technique that we used was fine-tuning a bert model which proved to be quite effective. a strong point of our system is its ability to get quite similar scores among both general and focused cases. this is likely due to our upsampling which helped with the largely imbalanced dataset. the one downside in our system seems to be the issue of cascading errors. this is reflected in the scores as we do part of subtask b first before moving on to subtask a and the ilo scores are much higher than our ari scores. moving forward we will need some way to eliminate the impact of these errors.