TAVT : Towards Transferable Audio-Visual Text Generation (2023)

Abstract: Audio-visual text generation aims to understand multi-modality contents and translate them into texts. Although various transfer learning techniques of text generation have been proposed, they focused on uni-modal analysis (e.g. text-to-text, visual-to-text) and lack consideration of multi-modal content and cross-modal relation. Motivated by the fact that humans can recognize the timbre of the same low-level concepts (e.g., footstep, rainfall, and laughing), even in different visual conditions, we aim to mitigate the domain discrepancies by audio-visual correlation. In this paper, we propose a novel Transferable Audio-Visual Text Generation framework, named TAVT, which consists of two key components: Audio-Visual Meta-Mapper (AVMM) and Dual Counterfactual Contrastive Learning (DCCL). (1) AVMM first introduces a universal auditory semantic space and drifts the domain-invariant low-level concepts into visual prefixes. Then the reconstruct-based learning encourages the AVMM to learn “which pixels belong to the same sound” and achieve audio-enhanced visual prefix. The well-trained AVMM can be further applied to uni-modal setting. (2) Furthermore, DCCL leverages the destructive counterfactual transformations to provide cross-modal constraints for AVMM from the perspective of feature distribution and text generation. (3) The experimental results show that TAVT outperforms the state-of-the-art methods across multiple domains (cross-datasets, cross-categories) and various modal settings (uni-modal, multi-modal).

## introduction audio-visual text generation bridges the gap between perception (visual and auditory) and communication (via language), and is hence becoming an increasingly important goal for artificial agents. uni-modal text generation tasks like machine translation @xcite @xcite figure 1 : examples of transferable multi-modal text generation in the source domain (real-word) and the target domain (animation). for the same events ("speaking", "cutting"), although the visual differences are significant, the sounds are similar. @xcite , and image caption @xcite @xcite have already flourished as a result of the large-scale pre-training and huge model capacity. however, for audiovisual text generation tasks, data annotation is more arduous (temporal structure) and expensive (requires monitors and speakers) than uni-modal text generation. moreover, despite the effectiveness, existing works @xcite @xcite inevitably suffer severe degradation due to varying construction conditions in different domains. in this paper, to break through the constraint, we propose a novel task, named transferable audiovisual text generation. the main challenge of this task is the multi-modal domain shifts caused by varying conditions, like the visual style and audio energy. one common approach to handle the domain shift is domain-alignment-based transfer learning. however, existing works @xcite @xcite focus on the uni-modal analysis, which is insufficient due to the lack of consideration of crossmodal relations. we observe that while audio and visual are often correlated in natural events and jointly affect human perception, they have different characteristics. as shown in figure 1 , the fact timbre is an intrinsic property of the object leads to sounds of the same concept ("speaking" or "cutting") being similar across domains where the appearance like background, perspective, and style is significantly different. based on this phenomenon, domain invariant low-level concepts can be extracted from the visual with the supervision of the audio, which is pervasive, reliable, and cheaper in contrast to expensive human annotation. ## related work audio-visual learning. in the past few years, there have been several works that focus on audiovisual learning. @xcite used a two-stream neural network to find the most similar visual area to the current audio clip. some works @xcite employed contrastive learning to match the visual and audio components. recently, @xcite proposed a framework for cross-modal representation learning with a discrete embedding space that was shared amongst different modalities and promoted model interpretability. the above approaches focus on the correlation of audio-visual pairs. while we aim to align the visual of different domains with a universal auditory semantic space and enhance multi-modal transfer learning. ## methods we aim to train a model that can learn and quickly adapt to new multimodal domains with limited labeled data under the meta-learning setting. in the next sections, we will first define the audio-visual meta-learning setting, then explain our architecture, and finally describe how it is used during training and inference time. ## conclusion in this paper, we first study the task of transferable audio-visual text generation tasks. to mitigate multimodal content domain shift, we observe that low-level visual concepts have similar sounds in different domains and propose a novel framework tavt with two technical contributions. the first one is the audio-visual meta-mapper, which can transfer the domain-invariant concept information within the universal auditory semantic space into the visual prefix. moreover, the well-trained audio-visual meta-mapper can also reconstruct semantic audio features in audio-absent mode. we then apply dual counterfactual contrastive learning to directly optimize the visual-audio alignment. extensive experiments on both cross-datasets and cross-domains benchmarks verify the effectiveness of our model. furthermore, our tavt framework can be transferred to other text generation tasks e.g., video qa in a plug-and-play fashion.