Text-centric Alignment for Bridging Test-time Unseen Modality (2025)

Abstract: This paper addresses the challenge of handling unseen modalities and dynamic modality combinations at test time with our proposed text-centric alignment method. This training-free alignment approach unifies different input modalities into a single semantic text representation by leveraging in-context learning with Large Language Models and uni-modal foundation models. Our method significantly enhances the ability to manage unseen, diverse, and unpredictable modality combinations, making it suitable for both generative and discriminative models to adopt on top. Our extensive experiments primarily evaluate on discriminative tasks, demonstrating that our approach is essential for LLMs to achieve strong modality alignment performance. It also surpasses the limitations of traditional fixed-modality frameworks in embedding representations. This study contributes to the field by offering a flexible and effective solution for real-world applications where modality availability is dynamic and uncertain.

## introduction this work targets the challenge of handling testtime unseen modalities and dynamic modality combinations for multimodal models where the input modality in the testing (or prediction) phase differs from that in training. the motivation for this research arises from the dynamic nature of real-world data, where modalities can unpredictably vary or even be absent at inference time. consider the following scenario: a hospital has extensive image and text data about its patients, such as x-ray images and doctors' written diagnoses. this data can be used to train an ai model that diagnoses patients based on both image and text inputs. however, to enhance patient satisfaction, the hospital wants to develop a dialog system that can diagnose figure 1 : this paper establishes a general method applicable to all mismatch types and combinations. as the figure shows, the model utilizes three modalities during training. our unified model can handle any combination of modalities during inference with in-context learning, including completely unseen modalities. additionally, it can deal with the situation that modality combinations dynamically change during inference, covering all possible combinations of both seen and unseen modalities. our objective is to develop a downstream model that is invariant across modalities. this model should be capable of being trained on data of certain modalities and performing zero-shot predictions on various modality combinations during testing, regardless of whether they were seen during training, as shown in figure 1 and figure 3 . note that simply converting all modalities into text for training and inference is insufficient, as the mismatch between different modalities does not translate and align as seamlessly in text, especially in complex scenarios involving multiple modality combinations. a brief summary of the results is shown in ## methodologies this section describes how tamml enables the generalization to unseen testing modalities and unseen modality combinations. we explicitly separate our pipeline into two phases: training-free alignment method, which includes modules that do not require any model training (sections 3.2.1 to 3.2.4), and downstream prediction, which requires minimal supervised training (section 3.3). figure 3 presents an overview of the entire process. in section 3.1, we define the problem setup and notations. sections 3.2.1-3.2.4 explain the components of the training-free alignment method: text transformation, text-style translation, modality summarization, and llm reasoning augmentation. section 3.3 describes the downstream task setup with minimal supervised training. ## computational cost and practicality analysis we report the average runtime latency for each component of tamml and compare it with existing baselines in ## conclusion and future directions our study has effectively harnessed large language models (llms) for multimodal learning, creating a unified semantic space that integrates various data modalities through text in a complete incontext learning manner. through techniques such as text transformation, text-style translation, summarization, and reasoning augmentation, we have demonstrated our proposed tamml alignment that the operations performed in the text domain using in-context learning with llms can achieve comparable performance to traditional methods operating in embedding space. this approach not only opens new avenues in multimodal learning but also underscores the significant potential and advantages of text as a unifying medium. future efforts will focus on refining tamml for broader multimodal tasks as well as other challenges in multimodal learning through text-centric approach, such as modality robustness, modality collapse and modality competition.