TILFA : A Unified Framework for Text, Image, and Layout Fusion in Argument Mining (2023)

Abstract: A main goal of Argument Mining (AM) is to analyze an author’s stance. Unlike previous AM datasets focusing only on text, the shared task at the 10th Workshop on Argument Mining introduces a dataset including both texts and images. Importantly, these images contain both visual elements and optical characters. Our new framework, TILFA (A Unified Framework for Text, Image, and Layout Fusion in Argument Mining), is designed to handle this mixed data. It excels at not only understanding text but also detecting optical characters and recognizing layout details in images. Our model significantly outperforms existing baselines, earning our team, KnowComp, the 1st place in the leaderboard of Argumentative Stance Classification subtask in this shared task.

## introduction argument mining (am) aims to automatically analyze the structure and components of arguments in text. persuasiveness analysis is a crucial aspect of it, which has gained significant attention in the nlp community @xcite . however, previous works focus solely on text, overlooking other modalities like images which can also impact an argument's persuasiveness. to fill this gap, @xcite introduces imagearg, a dataset going beyond text to include also images. it features tweets centered on contentious topics like gun control and abortion. these associated images contain not only objects but also optical characters (e.g., slogans, tables). the 10th workshop on argument mining in emnlp 2023 introduces a shared task @xcite called imagearg shared task 2023, centering around this dataset. it is divided into two subtasks: argumentative stance (as) classification and image persuasiveness (ip) classification. tweet text: background checks save lives. full stop. we deserve #morethanthoughtsandprayers from our elected leaders to end gun violence. topic: gun control stance: support tweet text: a child deserves a chance at life. a child deserves a future. a child deserves to be loved. we will always fight for the innocent unborn. #prolife topic: abortion stance: oppose tweet image: tweet image: we primarily focus on the former, which aims to identify the stance of a given tweet towards a specific topic. examples can be found in fig. 1 . after scrutinizing the dataset, we found several challenges: (1) imbalanced data distribution (ratio of positive to negative examples on abortion topic is about 1 : 2.65); (2) limited data size (neither of the two topics has more than 1000 entries); (3) presence of both objects and optical characters in images (they are difficult to be handled by a single model at the same time). to address these challenges, we have made the following contributions: data augmentation: data augmentation enhances a model's performance and increases its generalization capabilities in natural language processing (nlp). at the word level, wang et al. (2019) used databases like wordnet (miller, 1994), to replace certain words with their synonyms, while rizos et al. (2019) implemented embedding replacement to find contextually fitting words; at the document level, yu et al. (2018) used backtranslation, translating data to another language and then back to the source one. document ai: document ai refers to the extraction and comprehension of information from scanned documents, web pages, ads, posters, or images with textual content. previous works like hao et al. (2016); liu et al. (2019) all missed the integrated pre-training of text and layout details, which are vital for document image comprehension. to fix this, xu et al. (2020) proposed layoutlm. its updated versions, layoutlmv2 @xcite and layoutlmv3 @xcite , encapsulated text, layout and also image interactions within a unified multimodal framework. ## methods we employ back-translation @xcite to address data imbalance and apply wordnet @xcite for data augmentation, assisted by gloss-bert @xcite . our new framework, tilfa, uses deberta @xcite as the text encoder and layoutlmv3 @xcite as the image encoder, thus excels at not only understanding text but also detecting optical characters and recognizing layout details in images. we also experiment with several multimodal fusion mechanisms. consequently, we achieve the highest f1score in the argumentative stance classification subtask of the imagearg shared task 2023. ## conclusion we present tilfa, a new framework for multimodal argumentative stance classification. unlike existing methods, tilfa considers not only the text and images in tweets but also the characters and their layout information in those images. backtranslation and wordnet also contribute to our sota performance. our results reveal that better handling of images is essential to model improvement, and suggest that more effective methods for multimodal fusion are yet to be found.