M a CSC : Towards Multimodal-augmented Pre-trained Language Models via Conceptual Prototypes and Self-balancing Calibration (2024)

Abstract: Pre-trained language models (PLMs) that rely solely on textual data may exhibit limitations in multimodal semantics comprehension. Existing solutions attempt to alleviate this issue by incorporating explicit image retrieval or generation techniques.However, these methods: (1) focus exclusively on the static image modality; (2) inevitably encounter modality gaps and noise; (3) indiscriminately treat all modalities.In this paper, we propose a novel multimodal-augmented framework termed MaCSC, which can infuse multimodal semantics into PLMs and facilitate a self-balancing calibration of information allocation.Specifically, MaCSC obtains modal-specific conceptual prototypes from contrastive pre-training models (e.g., CLIP),and aggregates the intra- and inter-modal semantics of the conceptual prototype to enhance PLMs.In addition, we utilize a novel self-balancing contrastive loss to achieve multi-scale self-balancing calibration of multimodal information during fine-tuning PLMs.Experimental results show that MaCSC consistently improves the performance of PLMs across various architectures and scales, and outperforms competitive baselines on multiple NLP tasks.

## introduction large-scale pretrained language models (plms), (e.g., bert @xcite ,t5 @xcite , @xcite , etc.) have achieved great success on various natural language understanding (nlu) tasks. most plms undergo training on textual data alone without incorporating information from other modalities. those approaches contrast with human learning @xcite , which typically involves multiple modalities beyond just @xcite . textual data may not fully capture the general characteristics of entities @xcite , and word fre-* corresponding author. quency may not consistently reflect real-world likelihoods @xcite . consequently, these language learners may lack real-world knowledge @xcite and could suffer from human reporting bias @xcite . to address this issue, current research primarily focuses on augmenting plms by integrating visual information from either retrieved or synthesized images in nlp tasks @xcite . we classify these visually-augmented approaches for plms into three distinct categories: visually-augmented pre-training @xcite , visually-augmented finetuning @xcite , and training-free zero-shot visual enhancement @xcite . while these works represent notable advancements, we discover several limitations: (1) they focus exclusively on static image modality while neglecting other modal types like speech and video, which offer dynamic temporal information. in this paper, we propose a novel multimodalaugmented framework with conceptual prototypes enhancement and self-balancing calibration (macsc). macsc tackles the above issues through innovative strategies in the following two aspects: figure 2 shows the main differences between macsc and other methods. extensive experiments conducted on ten datasets across three nlp tasks (e.g., natural language understanding, q&a, and text generation tasks) demonstrate the effectiveness and universality of macsc. our contributions are: (1) we propose a novel multimodal-augmented framework called macsc, which integrates multimodal semantics into plms through conceptual prototypes and enhancement modules. (2) macsc utilizes a proposed novel self-balancing calibration strategy to balance multimodal information. (3) numerous experiments on multiple nlp tasks demonstrate the significant performance of macsc. ## methods in this section, we first introduce the task setting and overview of our method in section 2.1, and then describe our proposed conceptual prototype extraction approach in section 2.2 and crossmodal semantic enhancement in section 2.3. subsequently, we describe proposed allocation estimation strategies and self-balancing contrastive learning in section 2.4 and section 2.5, respectively. concat (b) semantic concept enhancement add & layernorm [input]: john finished the cake and drank the lemonade. modal-information enhancement modal-specific conceptual prototype self-balancing calibration classifier or regressor unfixed plm text encoder (bert) fix audio-ptm text encoder (aduioclip) fix image-ptm text encoder (clip) attention weight attention weight topk topk hold or rewrite for video-ptm text encoder (clip-vip) weighted logits aggregation predictions topk attention weight coarse-grained balancing conceptual prototype coarse-grained balancing fine-grained balancing (a) conceptual prototype extraction (c) self-balancing calibration vanilla pipeline our additional pipeline multimodal conceptual prototype ## related works pre-trained language models. recently, largescale pre-trained language models (plms) have demonstrated remarkable success through selfsupervised learning on extensive text corpus @xcite @xcite . by fine-tuning these plms, significant improvements are observed in downstream tasks including natural language processing (nlp), question answering, and text generation. previous work has shown that language learners trained solely on textual data can exhibit biases and a lack of multimodal understanding of the objective world @xcite @xcite @xcite . further-more, recent research suggests that these limitations are not mitigated by merely expanding the text corpus @xcite @xcite . in this paper, we propose a universal multimodal-augmented framework for enriching the integration of multimodal semantics for plms. multimodal contrastive pre-trained models. mc-ptms are trained on a substantial corpus of modality-pairing samples, mapping multiple different modalities to a unified representation space. clip @xcite performs pre-training on a large number of image text pairs, achieving alignment between image semantics and text semantics. audioclip @xcite extends clip to audio modality and clip-vip @xcite proposes a new strategy to transfer the image domain to the video domain. the latest work indicates that models similar to clip have the problem of modality gaps @xcite . here, we utilize mc-ptms to obtain text representations aligned with multimodal data as proxies, language models with additional modal enhancement. most studies consider injecting visual modality information into plms through retrieval or image generation. @xcite bansal (2020a ## conclusion in this paper, we propose a novel general multimodal-augmented framework with selfbalancing calibration for plms called macsc. macsc can efficiently inject other modal semantic information into plms using multimodal conceptual prototypes. in addition, macsc adopts a novel self-balancing contrastive loss to achieve multiscale self-balancing calibration of multimodal information during fine-tuning plms. experimental results show that macsc consistently improves the performance of plms across various architectures and scales on multiple nlp tasks.