Cross-Modal Discrete Representation Learning (2022)

Abstract: In contrast to recent advances focusing on high-level representation learning across modalities, in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words. Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities. Beyond the shared embedding space, we propose a Cross-Modal Code Matching objective that forces the representations from different views (modalities) to have a similar distribution over the discrete embedding space such that cross-modal objects/actions localization can be performed without direct supervision. We show that the proposed discretized multi-modal fine-grained representation (e.g., pixel/word/frame) can complement high-level summary representations (e.g., video/sentence/waveform) for improved performance on cross-modal retrieval tasks. We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities.

## introduction toddlers acquire much of their knowledge through grounded learning -visual concepts can be acquired through language, and language acquisition emerges through visual interaction. inspired by this type of grounded learning, a rich body of representation learning research @xcite @xcite @xcite has been exploring the potential to learn from multi-modal data such as video-text, video-audio, and image-audio pairs. these works typically focus on learning a joint embedding space between different modalities, in which high-level summary representations are extracted as embedding vectors. these embedding vectors often represent entire video clips, spoken utterances, or sentences as single vectors, and can be useful on tasks such as cross-modal data retrieval, e.g., finding the most similar visual scene according to a spoken language description. the predominant approach to learning these embedding vectors is to use modality-independent encoders, and while this has been successful for downstream retrieval tasks, it makes it difficult to compare the activations of the encoders from different modalities. further, the space of continuous embedding vectors is unbounded, which makes interpreting the learned representations challenging. to this end, we propose to jointly learn highlevel embedding vector representations with a finegrained discrete embedding space that is shared across different modalities. the discrete embedding space enables model interpretability since there are a finite number of embedding vectors which are shared across modalities. besides the shared embedding space, we propose a cross-modal code matching (cmcm) objective that guides the embedding space to capture cross-modal correspondences of concepts, actions, and words. this not only improves downstream performance on retrieval, but also allows us to better interpret what the model recognized through cross-modal grounded learning. to verify the effectiveness of our proposed learning framework, we conducted experiments in several cross-modal domains, including video-text, video-audio, and image-audio. we found consistent improvements over baseline models, verifying that the gain was not restricted to the particular choice of network architecture, input modalities, or dataset. we also demonstrate the interpretability of the fine-grained discrete representations by showing the cross-modal relations between the embedding vectors and semantic concepts appearing in the input modalities. our approach also enables cross-modal concept localization without requiring any labels during training. ## methodology figure 1 provides an overview of the proposed framework. we begin by describing the two-branch cross-modal representation learning paradigm in section 2.1 (the blue and yellow regions). next, we introduce our shared discrete embedding space in section 2.2 (the green region). finally, in section 2.3 and figure 2 , we introduce the cross-modal code matching objective which guides the model to learn semantically meaningful representations through the shared discrete embedding space. ## related work examples of the cross-modal learning paradigm. as described in section 2.1, many of the existing methods for cross-modal learning fit into the paradigm where encoders are modalityindependent. this paradigm has been shown to be effective by achieving state-of-the-art retrieval performance on benchmark datasets with the modality pairs that we considered in this work: videotext @xcite , videoaudio @xcite , and image-audio @xcite . while these prior works relied on different pre-training datasets, model architectures, and objective functions, they all leverage modalityindependent encoders. one of the most important features of this paradigm is the fixed inference time for retrieval. since the encoders are modalityindependent, embedding vectors for samples in a given modality can be computed without using any samples from the other modality. thus retrieval only involves computing the dot product between embedding vectors from two different modalities. as a consequence, these models are more flexible for large-scale retrieval, and the embedding vectors from each modality can be used independently for other downstream tasks. other cross-modal learning frameworks. in contrast to the aforementioned works, some methods leverage cross-modal relations within the encoders instead of using modality-independent encoders. this has been done with both cross-modal encoders @xcite and cross-modal attention mechanisms @xcite @xcite . however, the cross-modal interactions increase the complexity for retrieval since every instance of a specific modality must be used as input with every instance of another modality to obtain the embedding vectors. with m and n samples in the modalities respectively, this increases the complexity from the modality-independent approach from o(m + n) to o(mn). further, it also makes analysis of the embedding vectors from any individual modality challenging and inhibits single-modality downstream tasks. our proposed framework builds on the modality-independent approach to enable light-weight retrieval, but it also enables crossmodal interaction through our proposed codebook and cross-modal code matching objective. uncovering semantic-level correspondences. image-audio models have been shown to discover spoken words and visual objects without supervision through retrieval tasks @xcite @xcite , and the audio embedding vectors have been shown to cluster into word-like speech units @xcite @xcite . some work has studied the ability of video-audio models to relate spoken words to visual objects and actions in videos @xcite . however, none of these models incorporated a shared embedding space that enabled modality-invariant representations. vq units have been used in the audio encoder of an image-audio model @xcite , which allowed it to capture the hierarchical structure of spoken language. while our proposed framework is similar in that it also discretizes the audio sequence video-text / msr-vtt @xcite frozen-in-time @xcite 31.0 59.5 70.5 -----clip4clip-meanp @xcite 43.1 70.4 80.8 16.2 ----clip4clip-tightt @xcite 40.2 71.5 80.5 13.4 ----our baseline † 42.6 71.2 80.8 15.5 43.0 70.9 80.9 12.5 proposed 43.4 72.3 81.2 14.8 42.5 71.2 81.1 12.0 † existing model reproduced with lmms for fair comparison, see with vq units, our work differs significantly by capturing the cross-modal interactions between visual and audio inputs in the shared embedding space rather than solely capturing the tree structure of speech. further, besides image-audio data, our proposed framework can handle video-audio and video-text data. ## conclusion in this paper, we proposed a framework for crossmodal representation learning with a discrete embedding space that is shared amongst different modalities and enables model interpretability. we also propose a cross-modal code matching objective that encourages models to represent crossmodel semantic concepts in the embedding space. combining our discrete embedding space and objective with existing cross-modal representation learning models improves retrieval performance on video-text, video-audio, and image-audio datasets. we also analyze the shared embedding space and find that semantically related video and audio inputs tend to use the same codewords.