MAMK it: A Comprehensive Multimodal Argument Mining Toolkit (2024)

Abstract: Multimodal Argument Mining (MAM) is a recent area of research aiming to extend argument analysis and improve discourse understanding by incorporating multiple modalities. Initial results confirm the importance of paralinguistic cues in this field. However, the research community still lacks a comprehensive platform where results can be easily reproduced, and methods and models can be stored, compared, and tested against a variety of benchmarks. To address these challenges, we propose MAMKit, an open, publicly available, PyTorch toolkit that consolidates datasets and models, providing a standardized platform for experimentation. MAMKit also includes some new baselines, designed to stimulate research on text and audio encoding and fusion for MAM tasks. Our initial results with MAMKit indicate that advancements in MAM require novel annotation processes to encompass auditory cues effectively.

## introduction recent studies in argumentation analysis highlight the importance of including paralinguistic features in argumentative discourse analysis across various domains, including advertisements, news coverage, and legal analytics @xcite . similar considerations have been made for fake news detection @xcite . for these reasons, multimodal argument mining (mam) recently emerged as an extension of argument mining, aiming to validate these propositions empirically and gain a more comprehensive understanding of argumentative discourse by integrating multiple modalities. mam is a growing research field. the tasks addressed so far include argument detection, argument component classification, relation classification, and fallacy classification @xcite @xcite @xcite , 2024). however, despite these encouraging results, and similarly to what happens in other domains @xcite , the lack of standardized tools is hampering progress since mam researchers struggle to access and evaluate models and datasets. for one thing, mam resources are often hosted across various sites and repositories, each employing its own distinct methods for loading and reconstructing datasets and models. as a consequence, a fair model comparison may be problematic, which in turn limits the experimental evaluation of new models. we then introduce a pytorch toolkit tailored for mam. our toolkit, mamkit, currently includes 4 datasets and 6 models, providing researchers with a standardized platform for experimentation and evaluation. mamkit offers a simple interface to load, reconstruct and process existing datasets, and contribute new resources. moreover, all models within mamkit are implemented uniformly, facilitating seamless integration and comparison across tasks and datasets. to ensure the reproducibility and reliability of our models, all the resources and models in mamkit have been validated against the original research papers, offering a shared interface for benchmarking and model comparison. besides literature models, mamkit explores and integrates advanced audio encoding and fusion methods. indeed, previous research in mam has largely overlooked advanced audio encoding and fusion strategies @xcite , thus mamkit intends to present an opportunity to shed light on the significance of audio and the synergistic interaction between both modalities in argument mining tasks. ## related work we overview the mam literature and the landscape of toolkits built to address relevant ai tasks in different application domains. ## mamkit mamkit is an open-source, publicly available 1 pytorch toolkit designed to access and develop datasets, models, and benchmarks for mam. it provides a flexible interface for accessing and integrating datasets, models, and preprocessing strategies through composition or custom definition. mamkit is designed to be extendible, ensure replicability, and provide a shared interface as a common foundation for experimentation in the field. at the time of writing, mamkit offers 4 datasets and 6 distinct model architectures, along with audio and text processing capabilities, organized in 5 main components (see figure 1 ). ## experiments we employ mamkit to provide a robust and reproducible overview of a significant share of the work published on mam so far. in particular, we evaluate mamkit supported models on all available tasks and datasets. we build our evaluation as follows. regarding model evaluation, we compute macro f1-score except on ukdebates for which we report binary f1-score @xcite . we carry out a repeated five-fold cross-validation routine for ukdebates and m-arg γ using the same folds defined in @xcite . similarly, we perform a repeated train and test routine for mm-used on official data splits @xcite . we set the number of repetitions to three in both cases. lastly, we perform a leave-one-out routine for mm-used-fallacy @xcite . see appendix b for additional details. ## results ukdebates we observe no notable benefits in integrating the audio modality in all models, comparable to the results reported in @xcite . specifically, multimodal models show equal or lesser classification performance than their textonly modules. m-arg γ our results significantly differ from those reported in @xcite . in particular, ensemble and mul-ta, are noticeably underperforming compared to their text-only counterparts. the only exceptions are mm-bert and csa with slightly higher performance. additionally, audio-only models fail to learn the task. mm-used we observe a small performance gap between audio-only and text-only models, suggesting that the audio modality may be a valuable indicator in both asd and acc tasks. however, multimodal models achieve comparable performance to their text-only counterparts, with minor improvements only for mm-bert (+1.7), csa (+0.9), ensemble (+0.2) and mul-ta (+1.3) in asd, csa (+1.4), and mul-ta (+1.8) in acc. model ukdebates (asd) m-argγ (arc) mm-used (asd) mm-used (acc) mm-used-fallacy (afc) text only bilstm (t1) .552±.047 .120±.006 .811±.004 .663±.002 .525±.113 bert (t2) .654±.003 .132±.004 .824±.009 .679±.004 .594±.122 roberta (t3) .692±.005 .172±.015 .839±.010 .680±.001 .615±.097 audio only bilstm (a1) .393±.040 .024±.012 .774±.008 .596±.005 .657±.000 transformer (a2) .455±.004 .000±.000 .771±.019 .526±.004 .629±.162 text audio bilstm (t1 + a1) .533±.009 .084±.016 .815±.006 .667±.000 .572±.099 mm-bert (t2 + a1) .662±.004 .160±.015 .841±.005 .680±.004 .599±.128 mm-roberta (t3 + a1) .687±.010 .178±.012 .837±.009 .678±.003 .624±.074 csa (t2 + a2) .663±.014 .160±.015 .833±.011 .693±.001 .582±.114 ensemble (t2 + a2) .586±.015 .011±.011 .826±.011 .681±.002 .612±.134 mul-ta (t2 + a2) .616±.019 .098±.031 .837±.006 .697±.003 .605±.110 ## conclusion mam is a new, exciting and largely unexplored research domain with interesting applications. we believe that, at present, an open and collaborative standardized platform for experimentation and benchmarking has the potential to build a stronger community around it, that will be able to focus on the innovations needed to push the envelope. to this end, we developed an open-source pytorch toolkit named mamkit. mamkit offers several datasets, state-of-the-art models, and processing strategies. this paper introduces the platform and discusses some initial empirical results we obtained with it. remarkably, the advanced audio encoding and fusion techniques we introduced do not yield the performance improvement we hoped for. this result might be ascribed to weaknesses in the architectures, and motivate further research on novel encoding and fusion methods. however, the negative result might also be attributed to the fact that, in the available datasets, annotations were first made on the transcripts, and only later extended to the audio modality. as noted by @xcite , such a procedure does not exploit acoustic insights, hence it should be expected that the potential of mam architectures may not be fully leveraged, until datasets become available, that natively include auditory cues in the annotation process. this issue affects all mam datasets in mamkit, therefore a revision of the existing annotations would be required to effectively include auditory cues. in conclusion, further research is needed to understand audio characteristics better and devise methods to integrate them with textual annotations. that will necessitate collaboration across fields like argumentation and signal processing. mamkit could be a valuable resource for fostering such a collaboration. in a broader perspective, mamkit holds potential for further development and application, including its extension to additional modalities like images and video @xcite . for instance, we plan to incorporate the im-agearg dataset @xcite , which has been developed to address argument stance classification and image persuasiveness classification tasks. the imagearg dataset was notably extended during imagearg-2023 @xcite , the first shared task in mam, providing additional annotated samples. this dataset has been leveraged in various studies @xcite proposing diverse strategies for vision-language mam, thereby presenting an opportunity for integrating new models within mamkit. additionally, we plan to include in mamkit the mmclaims dataset @xcite , designed for multimodal claim detection in social media. furthermore, we aim to improve our understanding of multimodal discourse analysis and its practical implications through further experimentation with new datasets and by exploring transfer learning techniques to enhance model generalization across diverse domains.