Multi-task Learning in Argument Mining for Persuasive Online Discussions (2021)

Abstract: We utilize multi-task learning to improve argument mining in persuasive online discussions, in which both micro-level and macro-level argumentation must be taken into consideration. Our models learn to identify argument components and the relations between them at the same time. We also tackle the low-precision which arises from imbalanced relation data by experimenting with SMOTE and XGBoost. Our approaches improve over baselines that use the same pre-trained language model but process the argument component task and two relation tasks separately. Furthermore, our results suggest that the tasks to be incorporated into multi-task learning should be taken into consideration as using all relevant tasks does not always lead to the best performance.

## introduction argument mining (am) focuses on automatically identifying argumentative structures in text, and utilizing these structures in applications. am tasks include identifying argument components (e.g., "claim") and relations between them (e.g., "support"). however, most am studies have focused on monologues or micro-level models of arguments @xcite @xcite . am in dialogues and macro-level models have received less attention @xcite . in this study, we extend the work of @xcite in am for persuasive online discussions. particularly, we take advantage of a multi-task learning (mtl) approach to automatically identify the argument structures in persuasive dialogues that contain both micro-level and macrolevel argumentation. we identify argument components @xcite and two types of relations: intra-turn relations within one post and inter-turn relations across posts. our results demonstrate that using mtl improves the performance of both argument component and intra-turn/inter-turn relation classification. however, further analysis shows that the tasks in the mtl configuration should be chosen carefully depending on the focused task. we then try several techniques to increase the innate low precision of the relation classification tasks due to the highly imbalanced data, specifically @xcite and xgboost @xcite . our results demonstrate that smote is not very helpful but xgboost, when used with the representations learnt from mtl, can increase the precision and f-scores of the relation identification tasks. ## related work our work is closely related to @xcite . their system, called ampersand, tackles three am tasks on a dataset created from the change my view (cmv) subreddit foot_0 @xcite and focuses on transfer learning approaches with bert @xcite that take advantage of discourse and dialogue context. specifically, they define three separate tasks: argument component classification and intra/inter relation identification. for the first task, the requirement is to classify a given sentence into either claim, premise or non-argumentative. for the intra-relation identification task, given a pair of argumentative sentences from the same post, we need to answer if an argumentative relation between these two sentences exists. the inter-relation identification task is similar, except that the two sentences are from different posts. however, they treated the tasks of argument component classification and relation prediction separately and had independent bert models for the tasks. our approach works on the assumption that the three tasks are related to each other. many studies have shown that jointly learning several tasks during training usually leads to better performance in nlp problems @xcite @xcite . focusing on one single domain and dataset, @xcite treats am as a sequence tagging problem and uses sub-tasks such as component identification and relation classification as auxiliaries in mtl to improve performances. @xcite also formalizes argument component identification as a sequence tagging problem but utilizes multiple datasets from different domains in their mtl setup. they observe that the results on a small am dataset can be improved when other am datasets are leveraged as auxiliary tasks. these approaches, however, work on monologues where each data instance is from one person and therefore ignore the macro-structure of arguments. our work tackles am at dialogical level, specifically on online discussion forums. we hypothesize that mtl can help represent both micro and macro structure and use bert with a mtl setup to classify argument components and relations at the same time. ## data we use the same data from @xcite . they reuse the cmv corpus @xcite , where each sentence in a thread of the cmv subreddit is annotated as claim, premise or nonargumentative. additionally, they annotate the argument relation among these propositions (interturn/intra-turn) and expand the corpus by annotating additional argument components using the same guidelines. the final dataset consists of 112 threads with 2756 sentences. the proportions of claims, premises and non-argumentative components are 34%, 43% and 23% respectively. although several types of relations are annotated, the relation identification task only uses a binary label to represent if a relation exists between two components. the dataset is highly imbalanced in terms of relations, with only 4.6% of 27254 possible pairs having intra-turn and only 3.2% of 26695 having inter-turn relations, making low precision a major modeling challenge. below is an example of a discussion. user a makes a claim and supports it with a premise (intraturn relation). user b, however, disagrees with the reasoning made by user a (inter-turn relation). a: [i think the biggest threat to global stability comes from the political fringes.] 0:claim [it has been like that in the past.] 1:premise:support:0 b: [what happened in the past has nothing to do with the present] 2:attack:1 realizing that the data size is small, @xcite utilizes distant-labeled data and uses transfer learning for fine-tuning bert depending on the context. the imho+context dataset @xcite is used as micro-level context data. this is a corpus of opinionated claims in the form of sentences containing the internet acronyms imo (in my opinion) or imho (in my humble opinion) from reddit. the assumtion is that a relation exists between a sentence containing imho and the following one. for macro-level context data, they use the reddit quote feature and construct the qr dataset containing quote-response pairs. in reddit, when responding to a post, a user can quote another user's response and this feature is used to highlight what part of someone's argument a particular user is targeting in the cmv corpus. specifically, the qr dataset treats the quoted text and the following sentence as a positive inter-turn relation example. for a fair comparison, we also fine-tune bert using both distant datasets. ## methods we use ampersand's @xcite two relation classification constraints. for intra-turn relations, the source has to be a premise and the target can be a premise or a claim. for inter-turn relations, the source must be a claim. ## results using the same train/test split from ampersand (10% of the data for testing) @xcite , we compare our results with amper-sand. in tables 1, 2 and 3, xg stands for xg-boost, smo for smote, and refine for the refinement step from sec. 4.3. best results are in bold. to make the comparisons more consistent with our models, we applied refinement and xgboost on top of the final bert representation of amper-sand. the reported numbers from the second row of the tables are from our own rerun of ampersand and therefore they are slightly different from ones in the original paper. although we used the ampersand published code, the difference in pytorch version could be the cause for this discrepancy. the best results of ampersand from the original paper @xcite are reported in the first rows of the tables as ampersand*. ## qualitative analysis of intra-turn degradation using mtl_all one noticeable observation from section 5.2.1 and ## conclusion we show that using multi-task learning with micro and macro structures represented improves the performance of argumentative component classification and two relation prediction tasks, both with and without refinement. also, we observe that combining all tasks may not always be beneficial since we can have conflicts between some of them. further, our results demonstrate that using the xgboost model as the final classifier on top of the representation from bert, while not affecting the recall much, raises the precision scores for the intra-turn and inter-turn relation tasks. in sum, we achieve better results with mtl compared to the singletask training of @xcite , with and without our refinement and xgboost enhancements. future plans include leveraging contextual information to further improve performance and conducting further analyses on the incompatibility between intra-turn and inter-turn relation identification task in mtl.