In contrast to recent advances focusing on highlevel representation learning across modalities , in this work we present a self-supervised learning framework that is able to learn a representation that captures finer levels of granularity across different modalities such as concepts or events represented by visual objects or spoken words . Our framework relies on a discretized embedding space created via vector quantization that is shared across different modalities . Beyond the shared embedding space , we propose a Cross-Modal Code Matching objective that forces the representations from different views ( modalities ) to have a similar distribution over the discrete embedding space such that cross-modal objects / actions localization can be performed without direct supervision . We show that the proposed discretized multi-modal finegrained representation ( e.g. , pixel / word / frame ) can complement high-level summary representations ( e.g. , video / sentence / waveform ) for improved performance on cross-modal retrieval tasks . We also observe that the discretized representation uses individual clusters to represent the same semantic concept across modalities . Toddlers acquire much of their knowledge through grounded learning -visual concepts can be acquired through language , and language acquisition emerges through visual interaction . Inspired by this type of grounded learning , a rich body of representation learning research ( Harwath et al . , 2018 ; Miech et al . , 2020 ; Alayrac et al . , 2020 ; Monfort et al . , 2021 ; Luo et al . , 2021 ) has been exploring the potential to learn from multi-modal data such as video-text , video-audio , and image-audio pairs . These works typically focus on learning a joint embedding space between different modalities , in which high-level summary representations are extracted as embedding vectors . These embedding vectors often represent entire video clips , spoken utterances , or sentences as single vectors , and can be useful on tasks such as cross-modal data retrieval , e.g. , finding the most similar visual scene according to a spoken language description . The predominant approach to learning these embedding vectors is to use modality-independent encoders , and while this has been successful for downstream retrieval tasks , it makes it difficult to compare the activations of the encoders from different modalities . Further , the space of continuous embedding vectors is unbounded , which makes interpreting the learned representations challenging . To this end , we propose to jointly learn highlevel embedding vector representations with a finegrained discrete embedding space that is shared across different modalities . The discrete embedding space enables model interpretability since there are a finite number of embedding vectors which are shared across modalities . Besides the shared embedding space , we propose a Cross-Modal Code Matching ( CMCM ) objective that guides the embedding space to capture cross-modal correspondences of concepts , actions , and words . This not only improves downstream performance on retrieval , but also allows us to better interpret what the model recognized through cross-modal grounded learning . To verify the effectiveness of our proposed learning framework , we conducted experiments in several cross-modal domains , including video-text , video-audio , and image-audio . We found consistent improvements over baseline models , verifying that the gain was not restricted to the particular choice of network architecture , input modalities , or dataset . We also demonstrate the interpretability of the fine-grained discrete representations by showing the cross-modal relations between the embedding vectors and semantic concepts appearing in the input modalities . Our approach also enables cross-modal concept localization without requiring any labels during training . Figure 1 : An overview of the proposed framework . The proposed shared discrete embedding space ( green region , described in Section 2.2 ) is based on a cross-modal representation learning paradigm ( blue / yellow regions , described in Section 2.1 ) . The proposed Cross-Modal Code Matching L CMCM objective is detailed in Section 2.3 and Figure 2 . In this paper , we proposed a framework for crossmodal representation learning with a discrete embedding space that is shared amongst different modalities and enables model interpretability . We also propose a Cross-Modal Code Matching objective that encourages models to represent crossmodel semantic concepts in the embedding space . Combining our discrete embedding space and objective with existing cross-modal representation learning models improves retrieval performance on video-text , video-audio , and image-audio datasets . We also analyze the shared embedding space and find that semantically related video and audio inputs tend to use the same codewords .

Challenge: Existing multi-modal representation learning approaches aim to obtain high-level summary embeddings that are useful for retrieval but difficult to compare encoder activations from different modalities.
Approach: They propose a semi-supervised learning framework with a discrete embedding space shared across modalities by vector quantization and an objective to match representations between modalities.
Outcome: The proposed representations can complement high-level representations for retrieval and they use individual clusters to represent the same semantic concept across modalities demonstrating their interpretability.